9:07 a.m.
I have a fedora 13 box on which I have a remote mounted nfs share over a
fairly slow (10Mb/s) link. I'm then transferring data onto this share
from a different machine using scp.
The problem is that after scp reports that it's 100% complete the
program will hang for ~20 mins before it will move on to another file.
At this point it can't be killed.
It looks like the nfs daemon is caching write data (around 2GB of it)
which lets scp think its finished when actually there's loads of data
sitting in a write buffer. The hanging is presumably the time it takes
to flush the buffer (there is a process called nfsiod which is active
during this time and df shows data is still being written).
Does anyone know how to either make this buffer smaller, or get rid of
it all together so the scp can accruately report on its progress?
Thanks
Simon.
3:35 a.m.
On 30/09/2010 01:21, Wolfgang S. Rupprecht wrote:
Simon Andrews<simon.andrews(a)bbsrc.ac.uk> writes:
> Does anyone know how to either make this buffer smaller, or get rid of
> it all together so the scp can accruately report on its progress?
Why not cut to the chase and run scp to the remote server. TCP will do
a wonderful job of filling your pipe, but not too much.
I'd love to, but I don't control the remote end of the connection and
the only access I have is via an NFS mount.
6:31 a.m.
JB <jb.1234abcd <at> gmail.com> writes:
...
Firewall impact.
See 'man 5 nfs' - Mounting through a firewall.
Because you do not have access to your nfs server, the nfs client remains to
be checked for any messages coming from nfs server that may be blocked or
perhaps erroneously forwarded by your Fedora 13 client.
Your nfs server may be sending unsolicited NEW messages (as opposite to
ESTABLISHED, RELATED) that may be rejected by your (default) firewall settings.
You are also communicating with the source-of-the-big-file machine.
Iptables is your default netfilter/firewall on Fedora 13.
# iptables -L -n -v
If needed you should add extra log rules (man iptables; see LOG target; see
presumably /var/log/messages).
Test the firewall on your nfs client machine from an external machine.
JB
3:22 a.m.
On 29/09/2010 16:27, Ian Chapman wrote:
On 29/09/10 22:51, Simon Andrews wrote:
> That sounds like it might be an answer. It's a shame there's no way to
> specify this per-process, but this machine does have quite a bit of RAM
> in it, so having this specified as a percentage of RAM might make it
> larger than we'd want.
>
> I'll have a play and see what effect this has.
No problem, there's also a good page here which explains a bit more how
all this works, as well as some explanation on other tunables that might
suit you better.
http://www.westnet.com/~gsmith/content/linux-pdflush.htm
Thanks, that's really helpful.
1:11 a.m.
JB <jb.1234abcd <at> gmail.com> writes:
...
How nfsiod's number of read-ahead/write-behind worker threads is determined
and configured ?
$ cat linux-2.6.34.i686/fs/nfs/internal.h
/*
* NFS internal definitions
*/
...
/* Maximum number of readahead requests
* FIXME: this should really be a sysctl so that users may tune it to suit
* their needs. People that do NFS over a slow network, might for
* instance want to reduce it to something closer to 1 for improved
* interactive response.
*/
#define NFS_MAX_READAHEAD (RPC_DEF_SLOT_TABLE - 1)
...
$ cat linux-2.6.34.i686/include/linux/sunrpc/xprt.h
/*
...
* Declarations for the RPC transport interface.
...
*/
...
#define RPC_DEF_SLOT_TABLE (16U)
...
So, the max # of workers is 16 - 1 = 15
See: linux-2.6.34.i686/Documentation/kernel-parameters.txt
...
sunrpc.tcp_slot_table_entries=
sunrpc.udp_slot_table_entries=
[NFS,SUNRPC]
Sets the upper limit on the number of simultaneous
RPC calls that can be sent from the client to a
server. Increasing these values may allow you to
improve throughput, but will also increase the
amount of memory reserved for use by the client.
...
$ lsmod |grep -i sunrpc
sunrpc 163601 1
$ $ modinfo sunrpc
filename: /lib/modules/2.6.34.7-56.fc13.i686/kernel/net/sunrpc/sunrpc.ko
license: GPL
srcversion: 274F8EC6B56054A06EDF2A4
depends:
vermagic: 2.6.34.7-56.fc13.i686 SMP mod_unload 686
parm: min_resvport:portnr
parm: max_resvport:portnr
parm: tcp_slot_table_entries:slot_table_size
parm: udp_slot_table_entries:slot_table_size
$ ls /sys/module/sunrpc/parameters/
max_resvport pool_mode udp_slot_table_entries
min_resvport tcp_slot_table_entries
$ cat /sys/module/sunrpc/parameters/udp_slot_table_entries
16
$ cat /sys/module/sunrpc/parameters/tcp_slot_table_entries
16
Managing NFS and NIS, 2nd Edition.
By Hal Stern, Mike Eisler and Ricardo Labiaga
18.5. NFS async thread tuning.
...
If you are running eight NFS async threads on an NFS client, then the client
will generate eight NFS write requests at once when it is performing
a sequential write to a large file. The eight requests are handled by the NFS
async threads. ... when a Solaris process issues a new write requests while
all the NFS async threads are blocked waiting for a reply from the server,
the write request is queued in the kernel and the requesting process returns
successfully without blocking. The requesting process does not issue an RPC to
the NFS server itself, only the NFS async threads do. When an NFS async thread
RPC call completes, it proceeds to grab the next request from the queue and
sends a new RPC to the server. It may be necessary to reduce the number of NFS
requests if a server cannot keep pace with the incoming NFS write requests.
...
And for those who are not into NFS ...
Lesley Gore - It's My Party (1965)
http://www.youtube.com/watch?v=XsYJyVEUaC4
JB
7:21 p.m.
Simon Andrews <simon.andrews(a)bbsrc.ac.uk> writes:
Does anyone know how to either make this buffer smaller, or get rid
of
it all together so the scp can accruately report on its progress?
Why not cut to the chase and run scp to the remote server. TCP will do
a wonderful job of filling your pipe, but not too much.
-wolfgang
--
Wolfgang S. Rupprecht http://www.wsrcc.com/wolfgang/ (IPv6-only)
2:09 p.m.
JB <jb.1234abcd <at> gmail.com> writes:
Simon Andrews <simon.andrews <at> bbsrc.ac.uk> writes:
> ...
> I think it must be part of the kernel. There's no nfsiod binary
> anywhere on the system, but:
>
> $ ps aux | grep -i nfsiod
> root 8974 0.0 0.0 0 0 ? S Sep27 0:15 [nfsiod]
>
> Where the parent process is kthread.
>
> Simon.
Thanks.
Could you please give us (on Fedora 13) an unedited output of:
# cat /etc/fstab
# mount
JB
Well, actually these :-) :
$ man 8 mount
...
It is possible that files /etc/mtab and /proc/mounts don’t match. The
first file is based only on the mount command options, but the content
of the second file also depends on the kernel and others settings (e.g.
remote NFS server. In particular case the mount command may reports
unreliable information about a NFS mount point and the /proc/mounts
file usually contains more reliable information.)
# cat /etc/mtab
# cat /proc/mounts
JB
4:50 a.m.
On 29/09/2010 09:55, JB wrote:
Simon Andrews<simon.andrews<at> bbsrc.ac.uk> writes:
Hi Simon,
you know, we are Linux users from Missouri USA here ...
Can you tell us what system your nfs server is installed on, how its exported
nfs shares are configured ?
Then we can gain some valuable clues regarding performance of all factors
involved.
The server is an exported NFS share from a NetApp filer. Since the
server isn't maintained by us I can't provide specifics about how it's
configured.
The limiting factor seems to be the speed of the link between our client
and the server. We're actually operating pretty close to the available
wire speed so I don't think there's a problem with the setup in that
respect.
10:27 a.m.
On 29/09/10 22:51, Simon Andrews wrote:
That sounds like it might be an answer. It's a shame there's
no way to
specify this per-process, but this machine does have quite a bit of RAM
in it, so having this specified as a percentage of RAM might make it
larger than we'd want.
I'll have a play and see what effect this has.
No problem, there's also a good page here which explains a bit more how
all this works, as well as some explanation on other tunables that might
suit you better.
http://www.westnet.com/~gsmith/content/linux-pdflush.htm
--
Ian Chapman.
1:19 a.m.
On Thu, Sep 30, 2010 at 1:33 AM, Simon Andrews
<simon.andrews(a)bbsrc.ac.uk> wrote:
On 29/09/2010 20:09, JB wrote:
>> Thanks.
>> Could you please give us (on Fedora 13) an unedited output of:
> # cat /etc/mtab
> # cat /proc/mounts
remote.server.name:/vol/ftp/ftp-1 /mnt/remote nfs
rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=NNN.MMM.100.6,mountvers=3,mountport=4046,mountproto=tcp,addr=XXX.XXX.XXX.XXX
0 0
remote.server.name:/vol/ftp/ftp-1 /mnt/remote nfs
rw,tcp,addr=XXX.XXX.XXX.XXX 0 0
Do check with traceroute how packets are routed. It looks like the
server has multiple IP addresses. If you pick the wrong one
a straight shot route can be lost in preference to some other
out past the neighbors barn path.
--
NiftyFedora
T o m M i t c h e l l
1:13 a.m.
On Wed, Sep 29, 2010 at 8:27 AM, Ian Chapman
<packages(a)amiga-hardware.com> wrote:
On 29/09/10 22:51, Simon Andrews wrote:
> That sounds like it might be an answer. It's a shame there's no way to
> specify this per-process, but this machine does have quite a bit of RAM
> in it, so having this specified as a percentage of RAM might make it
> larger than we'd want.
>
> I'll have a play and see what effect this has.
No problem, there's also a good page here which explains a bit more how
all this works, as well as some explanation on other tunables that might
suit you better.
http://www.westnet.com/~gsmith/content/linux-pdflush.htm
--
If you have a lot of ram you can boot with a kernel flag
that specifies a smaller memory resource.
OR
You can write a program that gobbles a lot of memory
(compare malloc(); calloc()). Running a memory hog
can trigger IO of dirty pages out to disk or trigger
a the high/low water mark used to trigger IO. This works
because at the OS level all of free memory can be used
by the OS as a buffer of one type or another.
The nice thing about a littlepiggie program is that
you can run it from user space and also adjust
how much memory it grabs and when so you can tune
things to your liking. When you kill it you quickly
get all the memory back.
--
NiftyFedora
T o m M i t c h e l l
3:33 a.m.
On 29/09/2010 20:09, JB wrote:
> Thanks.
> Could you please give us (on Fedora 13) an unedited output of:
# cat /etc/mtab
# cat /proc/mounts
remote.server.name:/vol/ftp/ftp-1 /mnt/remote nfs
rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=149.155.100.6,mountvers=3,mountport=4046,mountproto=tcp,addr=XXX.XXX.XXX.XXX
0 0
remote.server.name:/vol/ftp/ftp-1 /mnt/remote nfs
rw,tcp,addr=XXX.XXX.XXX.XXX 0 0
9:25 p.m.
JB <jb.1234abcd <at> gmail.com> writes:
...
There are other things to consider.
$ man 5 exports
...
/etc/exports
...
General Options
...
async ...
sync ...
$ man 5 nfs
...
/etc/fstab
...
Valid options for either the nfs or nfs4 file system type
...
wsize=n The maximum number of bytes per network WRITE request
that the NFS client can send when writing data to a file
on an NFS server. ...
...
ac / noac Selects whether the client may cache file attributes.
...
Editor's Note:
The noac option is a mixture of a generic option, sync, and
an NFS-specific option actimeo=0. So it causes a significant performance
penalty.
But do not confuse file attributes caching with data caching.
...
The sync mount option
The NFS client treats the sync mount option differently than some other
file systems (refer to mount(8) for a description of the generic sync
and async mount options). If neither sync nor async is specified (or
if the async option is specified), the NFS client delays sending appli-
cation writes to the server until any of these events occur:
Memory pressure forces reclamation of system memory resources.
An application flushes file data explicitly with sync(2),
msync(2), or fsync(3).
An application closes a file with close(2).
The file is locked/unlocked via fcntl(2).
In other words, under normal circumstances, data written by an applica-
tion may not immediately appear on the server that hosts the file.
If the sync option is specified on a mount point, any system call that
writes data to files on that mount point causes that data to be flushed
to the server before the system call returns control to user space.
This provides greater data cache coherence among clients, but at a sig-
nificant performance cost.
Applications can use the O_SYNC open flag to force application writes
to individual files to go to the server immediately without the use of
the sync mount option.
...
NFS version 4 caching features
...
A file delegation ...
... Once a file has been delegated to a client, the client can cache
that file’s data and metadata aggressively without contacting the
server.
...
JB
8:10 p.m.
On Wed, 29 Sep 2010 17:21:25 -0700
Wolfgang S. Rupprecht wrote:
Why not cut to the chase and run scp to the remote server. TCP will
do
a wonderful job of filling your pipe, but not too much.
I see long delays with scp between two different systems talking
to ext3 filesystems on both ends. The scp command will say 100%,
but not actually exit, and the network monitor applet will show
lots of network traffic for quite a while. When the network traffic
dies down, the scp command finally exits.
I always figured it must be caching lots of packets in the 8 gig
of memory I have.
1:27 p.m.
Simon Andrews <simon.andrews <at> bbsrc.ac.uk> writes:
...
>
>> It looks like the nfs daemon is caching write data (around 2GB of it)
>> which lets scp think its finished when actually there's loads of data
>> sitting in a write buffer. The hanging is presumably the time it takes
>> to flush the buffer (there is a process called nfsiod which is active
>> during this time and df shows data is still being written).
> ...
...
$ ps aux | grep -i nfsiod
root 8974 0.0 0.0 0 0 ? S Sep27 0:15 [nfsiod]
Where the parent process is kthread.
Simon.
This is from HP Tru64 UNIX.
$ man 8 nfsiod
http://h30097.www3.hp.com/docs/base_doc/DOCUMENTATION/V51_HTML/MAN/MAN8/0...
JB
11:43 a.m.
On 09/29/2010 02:03 AM, Simon Andrews wrote:
On 29/09/2010 03:19, JD wrote:
>
>> <simon.andrews@bbsrc.ac.uk<mailto:simon.andrews@bbsrc.ac.uk>>
wrote:
>> The problem is that after scp reports that it's 100% complete the
>> program will hang for ~20 mins before it will move on to another file.
>> At this point it can't be killed.
> Hey! Simon,
> Listen: buffering is done by the filesystem internals
> in collaboration with the block io layer. Once the filesystem
> commits the write to block io layer, the write call returns to
> the calling program, and there is not an iota you can do about
> it!
I think I get the general process by which the caching happens, and I'm
not necessarily aiming to get rid of it, just set the cache size to
something which is appropriate for the speed of the link I'm operating over.
YOU CANNOT!!! IT IS DONE BY THE KERNEL!!! AND IT IS SET IN CONCRETE!!
READ MY SUBSEQUENT REPLY AS WELL.
> In the case of nfs, buffering is done by the nfsiod.
> Buffering will be done at both the server AND the client.
> This is especially noticeable when the nfs client writes onto
> and nfs mounted filesystem.
> nfsiod is the "helper" kernel thead. There will be as many of these
> as the admin configures the system for.
OK, so I'm the admin on the client system. How to I configure nfsiod?
You can
only set the NUMBER of how many to run.
But I have forgotten where it is done - perhaps in
some .conf file in /etc, or a file in /etc/sysconfig/
> Ditto with the main dispatcher,
> the nfsd process.
> The nfsiod is what buffers writes on the client side.
Which means that that is the part which is causing my problems. The
main problem is the disparity between the size of the cache on the
client (somewhere around 2GB) and the speed of transfer onto the NFS
mounted share (around 2MB/s). This means that every time the cache
needs to be flushed there is a ~20min wait during which the client is
completely blocked (so no chance to kill it or interact with it in any
way). Give me a 5MB cache and I'm a happy man
> If you want the scp to function more synchronously, you
> need to rewrite scp, so that it calls fsync after each write!
> This will force scp process to wait for the data to be flushed
> before the write call returns.
I don't need to be that draconian, just configure a sensible cache size.
I just can't see where I can set that.
Just give it up dude!
I told you already - cache amount CAN NOT BE CHANGED!!
Thanks for any advice
Simon.
10:29 a.m.
Simon Andrews <simon.andrews <at> bbsrc.ac.uk> writes:
...
I think it must be part of the kernel. There's no nfsiod binary
anywhere on the system, but:
$ ps aux | grep -i nfsiod
root 8974 0.0 0.0 0 0 ? S Sep27 0:15 [nfsiod]
Where the parent process is kthread.
Simon.
Thanks.
Could you please give us (on Fedora 13) an unedited output of:
# cat /etc/fstab
# mount
JB
9:51 a.m.
On 29/09/2010 14:12, Ian Chapman wrote:
On 28/09/10 22:07, Simon Andrews wrote:
> Does anyone know how to either make this buffer smaller, or get rid of
> it all together so the scp can accruately report on its progress?
You could try adjusting /proc/sys/vm/dirty_ratio.
"Contains, as a percentage of total system memory, the number of pages
at which a process which is generating disk writes will itself start
writing out dirty data"
The default seems to be 20 on Fedora 13. For example to set it to 5%
(which is the lowest it will go I believe):
That sounds like it might be an answer. It's a shame there's no way to
specify this per-process, but this machine does have quite a bit of RAM
in it, so having this specified as a percentage of RAM might make it
larger than we'd want.
I'll have a play and see what effect this has.
Cheers
Simon.
9:48 a.m.
On 29/09/2010 12:19, JB wrote:
Simon Andrews<simon.andrews<at> bbsrc.ac.uk> writes:
> ...
Hi,
thanks.
> It looks like the nfs daemon is caching write data (around 2GB of it)
> which lets scp think its finished when actually there's loads of data
> sitting in a write buffer. The hanging is presumably the time it takes
> to flush the buffer (there is a process called nfsiod which is active
> during this time and df shows data is still be
Your nfs client is a Fedora 13.
Can you tell me what nfsiod is and where is came from ?
Can you see it ?
$ ps aux | grep -i nfsiod
I think it must be part of the kernel. There's no nfsiosd binary
anywhere on the system, but:
$ ps aux | grep -i nfsiod
root 8974 0.0 0.0 0 0 ? S Sep27 0:15 [nfsiod]
Where the parent process is kthread.
Simon.
8:12 a.m.
On 28/09/10 22:07, Simon Andrews wrote:
Does anyone know how to either make this buffer smaller, or get rid
of
it all together so the scp can accruately report on its progress?
You could try adjusting /proc/sys/vm/dirty_ratio.
"Contains, as a percentage of total system memory, the number of pages
at which a process which is generating disk writes will itself start
writing out dirty data"
The default seems to be 20 on Fedora 13. For example to set it to 5%
(which is the lowest it will go I believe):
echo 5 > /proc/sys/vm/dirty_ratio
This should effectively lower the size of the buffer, but note this will
affect all file systems, not just NFS. However you may find that acceptable.
--
Ian Chapman.
6:19 a.m.
Simon Andrews <simon.andrews <at> bbsrc.ac.uk> writes:
...
Hi,
thanks.
It looks like the nfs daemon is caching write data (around 2GB of it)
which lets scp think its finished when actually there's loads of data
sitting in a write buffer. The hanging is presumably the time it takes
to flush the buffer (there is a process called nfsiod which is active
during this time and df shows data is still be
Your nfs client is a Fedora 13.
Can you tell me what nfsiod is and where is came from ?
Can you see it ?
$ ps aux | grep -i nfsiod
JB
4:03 a.m.
On 29/09/2010 03:19, JD wrote:
> <simon.andrews@bbsrc.ac.uk<mailto:simon.andrews@bbsrc.ac.uk>> wrote:
> The problem is that after scp reports that it's 100% complete the
> program will hang for ~20 mins before it will move on to another file.
> At this point it can't be killed.
Hey! Simon,
Listen: buffering is done by the filesystem internals
in collaboration with the block io layer. Once the filesystem
commits the write to block io layer, the write call returns to
the calling program, and there is not an iota you can do about
it!
I think I get the general process by which the caching happens, and I'm
not necessarily aiming to get rid of it, just set the cache size to
something which is appropriate for the speed of the link I'm operating over.
In the case of nfs, buffering is done by the nfsiod.
Buffering will be done at both the server AND the client.
This is especially noticeable when the nfs client writes onto
and nfs mounted filesystem.
nfsiod is the "helper" kernel thead. There will be as many of these
as the admin configures the system for.
OK, so I'm the admin on the client system. How to I configure nfsiod?
Ditto with the main dispatcher,
the nfsd process.
The nfsiod is what buffers writes on the client side.
Which means that that is the part which is causing my problems. The
main problem is the disparity between the size of the cache on the
client (somewhere around 2GB) and the speed of transfer onto the NFS
mounted share (around 2MB/s). This means that every time the cache
needs to be flushed there is a ~20min wait during which the client is
completely blocked (so no chance to kill it or interact with it in any
way). Give me a 5MB cache and I'm a happy man!
If you want the scp to function more synchronously, you
need to rewrite scp, so that it calls fsync after each write!
This will force scp process to wait for the data to be flushed
before the write call returns.
I don't need to be that draconian, just configure a sensible cache size.
I just can't see where I can set that.
Thanks for any advice
Simon.
3:55 a.m.
Simon Andrews <simon.andrews <at> bbsrc.ac.uk> writes:
I have a fedora 13 box on which I have a remote mounted nfs share over a
fairly slow (10Mb/s) link. I'm then transferring data onto this share
from a different machine using scp.
The problem is that after scp reports that it's 100% complete the
program will hang for ~20 mins before it will move on to another file.
At this point it can't be killed.
It looks like the nfs daemon is caching write data (around 2GB of it)
which lets scp think its finished when actually there's loads of data
sitting in a write buffer. The hanging is presumably the time it takes
to flush the buffer (there is a process called nfsiod which is active
during this time and df shows data is still being written).
Does anyone know how to either make this buffer smaller, or get rid of
it all together so the scp can accruately report on its progress?
Thanks
Simon.
Hi Simon,
you know, we are Linux users from Missouri USA here ...
Can you tell us what system your nfs server is installed on, how its exported
nfs shares are configured ?
Then we can gain some valuable clues regarding performance of all factors
involved.
JB
9:21 p.m.
On 09/28/2010 06:26 PM, Samuel Kidman wrote:
On Tue, Sep 28, 2010 at 10:07 PM, Simon Andrews
<simon.andrews(a)bbsrc.ac.uk <mailto:simon.andrews@bbsrc.ac.uk>> wrote:
I have a fedora 13 box on which I have a remote mounted nfs share
over a
fairly slow (10Mb/s) link. I'm then transferring data onto this share
from a different machine using scp.
The problem is that after scp reports that it's 100% complete the
program will hang for ~20 mins before it will move on to another file.
At this point it can't be killed.
It looks like the nfs daemon is caching write data (around 2GB of it)
which lets scp think its finished when actually there's loads of data
sitting in a write buffer. The hanging is presumably the time it
takes
to flush the buffer (there is a process called nfsiod which is active
during this time and df shows data is still being written).
Does anyone know how to either make this buffer smaller, or get rid of
it all together so the scp can accruately report on its progress?
Thanks
Simon.
Hey! Simon,
you could also re-write scp to open the file descriptor with
O_SYNC to force all writes to be synchronous, and you will
obviate the need to call fsync().
9:19 p.m.
On 09/28/2010 06:26 PM, Samuel Kidman wrote:
On Tue, Sep 28, 2010 at 10:07 PM, Simon Andrews
<simon.andrews(a)bbsrc.ac.uk <mailto:simon.andrews@bbsrc.ac.uk>> wrote:
I have a fedora 13 box on which I have a remote mounted nfs share
over a
fairly slow (10Mb/s) link. I'm then transferring data onto this share
from a different machine using scp.
The problem is that after scp reports that it's 100% complete the
program will hang for ~20 mins before it will move on to another file.
At this point it can't be killed.
It looks like the nfs daemon is caching write data (around 2GB of it)
which lets scp think its finished when actually there's loads of data
sitting in a write buffer. The hanging is presumably the time it
takes
to flush the buffer (there is a process called nfsiod which is active
during this time and df shows data is still being written).
Does anyone know how to either make this buffer smaller, or get rid of
it all together so the scp can accruately report on its progress?
Thanks
Simon.
Hey! Simon,
Listen: buffering is done by the filesystem internals
in collaboration with the block io layer. Once the filesystem
commits the write to block io layer, the write call returns to
the calling program, and there is not an iota you can do about
it! In the case of nfs, buffering is done by the nfsiod.
Buffering will be done at both the server AND the client.
This is especially noticeable when the nfs client writes onto
and nfs mounted filesystem.
nfsiod is the "helper" kernel thead. There will be as many of these
as the admin configures the system for. Ditto with the main dispatcher,
the nfsd process.
The nfsiod is what buffers writes on the client side.
If you want the scp to function more synchronously, you
need to rewrite scp, so that it calls fsync after each write!
This will force scp process to wait for the data to be flushed
before the write call returns.
8:26 p.m.
On Tue, Sep 28, 2010 at 10:07 PM, Simon Andrews
<simon.andrews(a)bbsrc.ac.uk>wrote:
I have a fedora 13 box on which I have a remote mounted nfs share
over a
fairly slow (10Mb/s) link. I'm then transferring data onto this share
from a different machine using scp.
The problem is that after scp reports that it's 100% complete the
program will hang for ~20 mins before it will move on to another file.
At this point it can't be killed.
It looks like the nfs daemon is caching write data (around 2GB of it)
which lets scp think its finished when actually there's loads of data
sitting in a write buffer. The hanging is presumably the time it takes
to flush the buffer (there is a process called nfsiod which is active
during this time and df shows data is still being written).
Does anyone know how to either make this buffer smaller, or get rid of
it all together so the scp can accruately report on its progress?
Thanks
Simon.
--
users mailing list
users(a)lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
This Article <http://nfs.sourceforge.net/nfs-howto/ar01s05.html> contains
information on how to adjust the buffer size of NFS and optimise file
transfers. Also scp has a -C option to enable compression.
12:39 p.m.
On 09/28/2010 07:07 AM, Simon Andrews wrote:
I have a fedora 13 box on which I have a remote mounted nfs share
over a
fairly slow (10Mb/s) link. I'm then transferring data onto this share
from a different machine using scp.
The problem is that after scp reports that it's 100% complete the
program will hang for ~20 mins before it will move on to another file.
At this point it can't be killed.
It looks like the nfs daemon is caching write data (around 2GB of it)
which lets scp think its finished when actually there's loads of data
sitting in a write buffer. The hanging is presumably the time it takes
to flush the buffer (there is a process called nfsiod which is active
during this time and df shows data is still being written).
Does anyone know how to either make this buffer smaller, or get rid of
it all together so the scp can accruately report on its progress?
Thanks
Simon.
I understand you mean your desktop nfs-mounts a directory
exported to your machine by a server over a 10 mb/s link.
I understaand you mean that you have write permissions to
write to this nfs mounted directory on your machine.
I understand that you scp from a third machine (again at 10 mb/s ???? -
you did not specify this part)
to the nfs mounted directory on YOUR machine.
Just to get the data from the third machine to yours before it
is even sent to the nfs server):
On a 10megabit/s link, only 80% of which is payload data,
transferring 2GB (I assume you mean binary GB):
.80 * 10000000 = 8000000 bits/s payload data
8000000 / 8 = 100000 bytes/s payload data
2147483648 / 1000000 = 2147.48 number of seconds it takes to download 2 GB.
2147.48 /60.00 = 35.79 minutes to download 2GB.
So, that's close enough to your 20 minutes download time.
I am being a bit pessimistic here as to how much of the ether bandwirdth
is used
for payload data.
At 90%, the transfer time comes down to 31 minutes.
So, 20 minutes is absolutely miraculous!!!
Be HAPPY!!
9:57 a.m.
On Tue, 28 Sep 2010 15:07:25 +0100
Simon Andrews wrote:
Does anyone know how to either make this buffer smaller, or get rid
of
it all together so the scp can accruately report on its progress?
I don't know how to fix it, but I don't think it has anything to
do with NFS. I see the same with ext3 disks at both ends.
4948
days inactive
4952
days old
31 comments
11 participants
participants (11)
-
Alan Cox -
Gordon Messmer -
Ian Chapman -
JB -
JD -
Kwan Lowe -
Nifty Fedora Mitch -
Samuel Kidman -
Simon Andrews -
Tom Horsley -
Wolfgang S. Rupprecht