Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Runtime Error in plugin.py #22

Open
papiveron opened this issue Dec 18, 2015 · 9 comments
Open

Runtime Error in plugin.py #22

papiveron opened this issue Dec 18, 2015 · 9 comments

Comments

@papiveron
Copy link

Hi,

I succeeded to install and run the plugin but I'm getting runtime error, the same error all time on a bad key ALL in the plugin.py module. Here is what is writen in log file :

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/newrelicnfs/plugin.py", line 178, in _get_nfs_stat_for
    volnfsstat = self.nfs_stats[volume]
KeyError: 'ALL'
2015-12-18 16:50:05,698 [ERROR] newrelicnfs.plugin:_get_nfs_stat_for: 'ALL'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/newrelicnfs/plugin.py", line 178, in _get_nfs_stat_for
    volnfsstat = self.nfs_stats[volume]
KeyError: 'ALL'
2015-12-18 16:51:06,216 [ERROR] newrelicnfs.plugin:_get_nfs_stat_for: 'ALL'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/newrelicnfs/plugin.py", line 178, in _get_nfs_stat_for
    volnfsstat = self.nfs_stats[volume]
KeyError: 'ALL'
2015-12-18 16:52:06,731 [ERROR] newrelicnfs.plugin:_get_nfs_stat_for: 'ALL'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/newrelicnfs/plugin.py", line 178, in _get_nfs_stat_for
    volnfsstat = self.nfs_stats[volume]
KeyError: 'ALL'
2015-12-18 16:53:07,248 [ERROR] newrelicnfs.plugin:_get_nfs_stat_for: 'ALL'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/newrelicnfs/plugin.py", line 178, in _get_nfs_stat_for
    volnfsstat = self.nfs_stats[volume]
KeyError: 'ALL'

Is this a bug in the code? I'm using the last release with all requirements.

My /proc/self/mountstats

[admin@ip-10-4-16-11]/t/n/n/d/newrelic-nfsiostat-0.2.5 ❯❯❯ cat /proc/self/mountstats
device rootfs mounted on / with fstype rootfs
device sysfs mounted on /sys with fstype sysfs
device proc mounted on /proc with fstype proc
device udev mounted on /dev with fstype devtmpfs
device devpts mounted on /dev/pts with fstype devpts
device tmpfs mounted on /run with fstype tmpfs
device /dev/disk/by-uuid/2dbe52e8-a50b-45d9-a2ee-2c240ab21adb mounted on / with fstype ext4
device tmpfs mounted on /run/lock with fstype tmpfs
device tmpfs mounted on /run/shm with fstype tmpfs
device /dev/xvdf mounted on /mnt/data with fstype ext4
device rpc_pipefs mounted on /var/lib/nfs/rpc_pipefs with fstype rpc_pipefs
device nfsd mounted on /proc/fs/nfsd with fstype nfsd

Thank for help.

Regards.
@TJM
Copy link
Member

TJM commented Dec 21, 2015

While there is probably an error in the code, I don't see any nfs mounted filesystems in your /proc/self/mountstats, so there is nothing to report on. I am guessing that its trying to report on something that isn't there and that is the cause of the error?

It just occurred to me that you may be trying to run this on the NFS Server, while it is designed to run "nfsiostat" against mounted NFS filesystems on the client side.

@papiveron
Copy link
Author

Hi,

Yes, you may be right, in fact I installed the plugin on the NFS server.

I understand now. So are you saying this client is for NFS client, and there is nothing to do in the server side?

And finally yes if the plugin is supposed to run on the client side, then you're right it should display an error saying it's on the server and should be run on the client, or something like that.

Thank you so much for your reply. I'll move to the client and let you know if it works.

Regards.

@papiveron
Copy link
Author

Hi,

Yes you were right. It works better on the client (I don't have the same error) and I can informations (empty) in the newrelic dashboard, But now I have another code error in the log file :

2015-12-22 10:53:10,483 [ERROR] newrelicnfs.plugin:_get_nfs_stat_for: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/newrelicnfs/plugin.py", line 191, in _get_nfs_stat_for
    op_prefix + '/Operations[ops/second]': op_stat[0],
TypeError: 'NoneType' object has no attribute '__getitem__'
2015-12-22 10:53:10,484 [ERROR] newrelicnfs.plugin:_get_nfs_stat_for: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/newrelicnfs/plugin.py", line 191, in _get_nfs_stat_for
    op_prefix + '/Operations[ops/second]': op_stat[0],
TypeError: 'NoneType' object has no attribute '__getitem__'
2015-12-22 10:53:10,484 [ERROR] newrelicnfs.plugin:_get_nfs_stat_for: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/newrelicnfs/plugin.py", line 191, in _get_nfs_stat_for
    op_prefix + '/Operations[ops/second]': op_stat[0],
TypeError: 'NoneType' object has no attribute '__getitem__'
2015-12-22 10:54:11,005 [ERROR] newrelicnfs.plugin:_get_nfs_stat_for: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/newrelicnfs/plugin.py", line 191, in _get_nfs_stat_for
    op_prefix + '/Operations[ops/second]': op_stat[0],
TypeError: 'NoneType' object has no attribute '__getitem__'
2015-12-22 10:54:11,006 [ERROR] newrelicnfs.plugin:_get_nfs_stat_for: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/newrelicnfs/plugin.py", line 191, in _get_nfs_stat_for
    op_prefix + '/Operations[ops/second]': op_stat[0],
TypeError: 'NoneType' object has no attribute '__getitem__'
2015-12-22 10:54:11,006 [ERROR] newrelicnfs.plugin:_get_nfs_stat_for: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/newrelicnfs/plugin.py", line 191, in _get_nfs_stat_for
    op_prefix + '/Operations[ops/second]': op_stat[0],
TypeError: 'NoneType' object has no attribute '__getitem__'

The screenshot show newrelic event dashboard, the instance ip-10-4-16-11 represent the server, and is not important as I stopped the service there.

nfs_events

Here is the client /proc/self/mountstats

device rootfs mounted on / with fstype rootfs
device sysfs mounted on /sys with fstype sysfs
device proc mounted on /proc with fstype proc
device udev mounted on /dev with fstype devtmpfs
device devpts mounted on /dev/pts with fstype devpts
device tmpfs mounted on /run with fstype tmpfs
device /dev/disk/by-uuid/2dbe52e8-a50b-45d9-a2ee-2c240ab21adb mounted on / with fstype ext4
device tmpfs mounted on /run/lock with fstype tmpfs
device tmpfs mounted on /run/shm with fstype tmpfs
device rpc_pipefs mounted on /var/lib/nfs/rpc_pipefs with fstype rpc_pipefs
device nfs-server:/mnt/data/home/my_app/share/var mounted on /var/www/my_app/site_0_5/var with fstype nfs4 statvers=1.0
        opts:   rw,vers=4,rsize=524288,wsize=524288,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,prot
        age:    3955058
        caps:   caps=0x7fff,wtmult=512,dtsize=32768,bsize=0,namlen=255
        nfsv4:  bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3,pnfs=not configured
        sec:    flavor=1,pseudoflavor=1
        events: 15611650 411326471 5778125 7750006 290994 112404 459171381 365619 532 5760917 185762 65348479 589104 86
        bytes:  96848469072 411857314 0 0 5868101312 411954367 6931753 185762 
        RPC iostats version: 1.0  p/v: 100003/4 (nfs)
        xprt:   tcp 816 0 53 0 9 76523467 76523467 0 82694865 0
        per-op statistics
                NULL: 0 0 0 0 0 0 0 0
                READ: 5761636 5761636 0 991001392 6226979548 25536 2701468 2766415
               WRITE: 86988 86988 0 428723976 11482416 2323 456265 459802
              COMMIT: 258 258 0 44376 31992 1 8642 8648
                OPEN: 13642503 13642503 0 3856097620 8021787844 2634925 6637257 9379588
        OPEN_CONFIRM: 6424732 6424732 0 1053656048 436881776 27094 2983107 3045838
         OPEN_NOATTR: 0 0 0 0 0 0 0 0
        OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0
               CLOSE: 13637816 13637816 0 2454806880 1800191712 2531266 6509936 9141195
             SETATTR: 868525 868525 0 235945380 236148056 1544 2810627 2819810
 FSINFO: 1 1 0 136 108 0 0 0
               RENEW: 0 0 0 0 0 0 0 0
         SETCLIENTID: 0 0 0 0 0 0 0 0
        SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
                LOCK: 5628139 5628139 0 1283215692 382713548 46170 2561006 2639964
               LOCKT: 0 0 0 0 0 0 0 0
               LOCKU: 5628133 5628133 0 1035576472 382713044 21745 2543548 2597473
              ACCESS: 2085219 2085221 0 341170236 558839016 6299 1120010 1173218
             GETATTR: 15611652 15611654 0 2434949552 3934132400 31966 7347991 7603312
              LOOKUP: 1382195 1382195 0 277148732 398921740 2456 668475 681692
         LOOKUP_ROOT: 0 0 0 0 0 0 0 0
              REMOVE: 1504 1504 0 325504 420892 2 812 825
              RENAME: 103 103 0 28704 54796 0 62 62
                LINK: 0 0 0 0 0 0 0 0
             SYMLINK: 0 0 0 0 0 0 0 0
              CREATE: 92 92 0 20648 50880 0 236 237
            PATHCONF: 1 1 0 132 72 0 0 0
              STATFS: 2 2 0 312 232 0 0 0
            READLINK: 0 0 0 0 0 0 0 0
             READDIR: 111548 111548 0 20073768 390562760 147 63025 63792
         SERVER_CAPS: 5 5 0 668 440 0 2 2
         DELEGRETURN: 0 0 0 0 0 0 0 0
              GETACL: 0 0 0 0 0 0 0 0
              SETACL: 0 0 0 0 0 0 0 0
        FS_LOCATIONS: 0 0 0 0 0 0 0 0
        RELEASE_LOCKOWNER: 5628133 5628133 0 720401024 247637852 18276 2628600 2676388
             SECINFO: 0 0 0 0 0 0 0 0
         EXCHANGE_ID: 0 0 0 0 0 0 0 0
        CREATE_SESSION: 0 0 0 0 0 0 0 0
        DESTROY_SESSION: 0 0 0 0 0 0 0 0
SEQUENCE: 0 0 0 0 0 0 0 0
        GET_LEASE_TIME: 0 0 0 0 0 0 0 0
        RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0
           LAYOUTGET: 0 0 0 0 0 0 0 0
        GETDEVICEINFO: 0 0 0 0 0 0 0 0
        LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
        LAYOUTRETURN: 0 0 0 0 0 0 0 0
        SECINFO_NO_NAME: 0 0 0 0 0 0 0 0
        TEST_STATEID: 0 0 0 0 0 0 0 0
        FREE_STATEID: 0 0 0 0 0 0 0 0
        GETDEVICELIST: 0 0 0 0 0 0 0 0

device nfs-server:/mnt/data/home/my_app/share/media mounted on /var/www/my_app/site_0_5/media with fstype nfs4 statvers
        opts:   rw,vers=4,rsize=524288,wsize=524288,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,prot
        age:    3955058
        caps:   caps=0x7fff,wtmult=512,dtsize=32768,bsize=0,namlen=255
        nfsv4:  bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3,pnfs=not configured
        sec:    flavor=1,pseudoflavor=1
        events: 15611650 411326471 5778125 7750006 290994 112404 459171381 365619 532 5760917 185762 65348479 589104 86
        bytes:  96848469072 411857314 0 0 5868101312 411954367 6931753 185762 
        RPC iostats version: 1.0  p/v: 100003/4 (nfs)
        xprt:   tcp 816 0 53 0 9 76523467 76523467 0 82694865 0
        per-op statistics
                NULL: 0 0 0 0 0 0 0 0
                READ: 5761636 5761636 0 991001392 6226979548 25536 2701468 2766415
               WRITE: 86988 86988 0 428723976 11482416 2323 456265 459802
              COMMIT: 258 258 0 44376 31992 1 8642 8648
                OPEN: 13642503 13642503 0 3856097620 8021787844 2634925 6637257 9379588
        OPEN_CONFIRM: 6424732 6424732 0 1053656048 436881776 27094 2983107 3045838
         OPEN_NOATTR: 0 0 0 0 0 0 0 0
         OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0
               CLOSE: 13637816 13637816 0 2454806880 1800191712 2531266 6509936 9141195
             SETATTR: 868525 868525 0 235945380 236148056 1544 2810627 2819810
              FSINFO: 1 1 0 136 108 0 0 0
               RENEW: 0 0 0 0 0 0 0 0
         SETCLIENTID: 0 0 0 0 0 0 0 0
        SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
                LOCK: 5628139 5628139 0 1283215692 382713548 46170 2561006 2639964
               LOCKT: 0 0 0 0 0 0 0 0
               LOCKU: 5628133 5628133 0 1035576472 382713044 21745 2543548 2597473
              ACCESS: 2085219 2085221 0 341170236 558839016 6299 1120010 1173218
             GETATTR: 15611652 15611654 0 2434949552 3934132400 31966 7347991 7603312
              LOOKUP: 1382195 1382195 0 277148732 398921740 2456 668475 681692
              LOOKUP_ROOT: 0 0 0 0 0 0 0 0
              REMOVE: 1504 1504 0 325504 420892 2 812 825
              RENAME: 103 103 0 28704 54796 0 62 62
                LINK: 0 0 0 0 0 0 0 0
             SYMLINK: 0 0 0 0 0 0 0 0
              CREATE: 92 92 0 20648 50880 0 236 237
            PATHCONF: 1 1 0 132 72 0 0 0
              STATFS: 2 2 0 312 232 0 0 0
            READLINK: 0 0 0 0 0 0 0 0
             READDIR: 111548 111548 0 20073768 390562760 147 63025 63792
         SERVER_CAPS: 5 5 0 668 440 0 2 2
         DELEGRETURN: 0 0 0 0 0 0 0 0
              GETACL: 0 0 0 0 0 0 0 0
              SETACL: 0 0 0 0 0 0 0 0
        FS_LOCATIONS: 0 0 0 0 0 0 0 0
        RELEASE_LOCKOWNER: 5628133 5628133 0 720401024 247637852 18276 2628600 2676388
             SECINFO: 0 0 0 0 0 0 0 0
         EXCHANGE_ID: 0 0 0 0 0 0 0 0
        CREATE_SESSION: 0 0 0 0 0 0 0 0
        DESTROY_SESSION: 0 0 0 0 0 0 0 0
            SEQUENCE: 0 0 0 0 0 0 0 0
        GET_LEASE_TIME: 0 0 0 0 0 0 0 0
        RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0
           LAYOUTGET: 0 0 0 0 0 0 0 0
        GETDEVICEINFO: 0 0 0 0 0 0 0 0
        LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
        LAYOUTRETURN: 0 0 0 0 0 0 0 0
        SECINFO_NO_NAME: 0 0 0 0 0 0 0 0
        TEST_STATEID: 0 0 0 0 0 0 0 0
        FREE_STATEID: 0 0 0 0 0 0 0 0
        GETDEVICELIST: 0 0 0 0 0 0 0 0

Thanks for help.

Regards.

@TJM
Copy link
Member

TJM commented Dec 23, 2015

just as a test (a guess), can you try using NFSv3, in your mount options you have vers=4, try letting that be vers=3 to see if that helps. I realize you may want to use NFSv4, but the stats may be different. I think someone was working on a fork for NFSv4 (#21) that you can maybe try. I need to test that against NFSv3 mounts and get it merged :)

@papiveron
Copy link
Author

Ok @TJM,

I'll test all that outside production environment and let you know.

Thanks.
Regards.

@papiveron
Copy link
Author

Hi @TJM,

I've setup a test client, and use nfs version 3 on it, and it seems to work better, but I can't really say it works till you confirm with the information I will give you below.

So now, logs look like this :

2015-12-24 12:16:36,146 [WARNING] newrelicnfs.plugin:__init__: Our Special Runner
2015-12-24 12:21:23,981 [WARNING] newrelicnfs.plugin:__init__: Our Special Runner
2015-12-24 12:22:05,833 [WARNING] newrelicnfs.plugin:__init__: Our Special Runner
2015-12-24 12:22:13,777 [WARNING] newrelicnfs.plugin:__init__: Our Special Runner
2015-12-24 12:22:15,535 [WARNING] newrelicnfs.plugin:__init__: Our Special Runner
2015-12-24 12:23:52,217 [WARNING] newrelicnfs.plugin:__init__: Our Special Runner

Nothing else.

And in the newrelic nfs plugin newrelic dashboard, I have the value 0 in every column like this fisrt screen-shot shows
nfs_stats_dashboard

For me it seems like I have no data, but when I click on any row there I have the attached graph (I split in two parts for scrolling issue)

nfs_overview_1
nfs_overview_2

Can you tell me please if it's normal?

I'll then dive into other plugin for version4 and test it after your answer.

Thanks.

Regards.

@papiveron
Copy link
Author

Here attached is the output of the command nfsiostat 5 10 -adps I ran on the client a few hours ago.
nfsiostats_report.txt

Let me know what you think about the dashboard please.

Regards.

@papiveron
Copy link
Author

Hi,

Yes the fix for NFSv4 support also works. Now I've only this in log file as mentionned above :

"2015-12-28 14:01:38,405 [WARNING] newrelicnfs.plugin:__init__: Our Special Runner
2015-12-28 14:01:40,139 [WARNING] newrelicnfs.plugin:__init__: Our Special Runner

But I've data in the dashboard for both version3 and vercion4.

Thanks.
Regards.

@TJM
Copy link
Member

TJM commented Dec 29, 2015

Yea, so that should be a "DEBUG" log level message that you are seeing there, I think I may have been trying to figure out what was wrong with the runner and set it to warning, then forgot to set it back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@TJM @papiveron