 |
FC++
v0.9.0-9e9b65
FileCatalyst Fast File Transfers - C++ Library
|
|
|
FC++
v0.9.0-9e9b65
FileCatalyst Fast File Transfers - C++ Library
|
|
The command-line tool fccl (or fccl.exe on Windows) uses the same FC++ library that is shipped as part of the FC++ SDK API. Prior to attempting to use the FC++ SDK API in your application, it is strongly recommended that you try a file transfer with fccl to understand the different file transfer options and their impact.
To get started, this simple command will download a file from a FileCatalyst server:
While this is enough to get started, it is not the most optimal transfer. Let's add a few additional parameters to understand what happens during a FC file transfer.
The –showstats parameter will display some additional statistics after the transfer has completed. We'll use those statistics to understand the impact of several other parameters.
One of the first things we normally want to control is the speed at which a transfer will take place. This is done with the –bandwidth parameter:
However, if you try this on a large enough file, or across a busy network such as the internet versus an internal network, you'll note you didn't get the rate you requested. This is because FC++ gradually ramps up to the specified speed. If you want to force FC++ to use a specific speed, you'll also want to specify a starting rate indentical to the desired maximum rate:
Part of the statistics displayed when –showstats has been specified lists the number of duplicate packets as well as the number of re-requested packets. Duplicate packets are not a big concern unless the number is extremely high. But re-requested packets indicates a problem that can result in poor file transfer performance. The goal is normally to get this number close to 0%. Here is an example of what that output can look like:
The 1751 retransmit requests in this example means the receiver had to ask the sender 1751 times to re-send a different part of the file because it was lost in transit. There are several possible reasons why this happens:
The easiest way to alleviate this problem is to turn on the FileCatalyst UDP congestion control. This allows the sender to slow down when the receiver detects that some packets have been dropped.
–aggression parameter to indicate how quickly it should try to recover when dropped UDP packets have been detected.By default, UDP download uses a single thread within FC++. This helps to limit the amount of memory needed while downloading, but it also limits the speed at which transfers can happen. If too many UDP packets arrive while the single thread is busy processing a previous packet, some of the newly-arrived UDP packets will be dropped by the operating system before FC++ can process them.
This problem can be addressed by enabling multi-threaded downloads. You'll also need to specify the number of threads which FC++ will use. Try with 3 to 5 threads to see if it makes a difference. Having threads also introduces lock contention, so there comes a point where adding more threads doesn't help.
When adding threads, it also helps to increase the number of file encoders on the server. The FC++ API tells the server the number of encoders to use with the –numencoders parameter. It is a good idea to have 1 or 2 more encoders on the server than the number of threads on the FC++ client.
Files transferred using FileCatalyst are broken up into parts called blocks and units. Every block is made up of 1 or more unit, and in typical usage, a single unit also happens to fit within a single UDP packet. The default block size is 4 MiB in size, and the default unit size is 1024 bytes.
The block size is typically not a problem, but the unit size can be further optimized. If you know the MTU between the FC server and the FC++ client, you can increase (or decrease) the unit size. With a typical network MTU of 1500 bytes, and taking into account the overhead that FileCatalyst needs, this means you can put more than the default 1024 data payload into each packet. For example:
In the above example, the block size was decreased from 4 MiB to 256 KiB, and the unit size was increased from 1024 bytes to 1300 bytes. But 1300 doesn't cleanly divide 256 KiB, and the size of the download file also isn't a perfect multiple of 1300. So FC++ automatically reduces both the block size and the unit size as necessary so the file can be transfered with a minimal amount of internal padding. In this example, 37 blocks of ≈138 KiB were downloaded, and a total of only 21 bytes of internal padding was necessary on the very last block.
MD5 checksum verification can be enabled to ensure the transfer wasn't corrupted. This runs after the transfer has finished, and can take non-trivial time to run on large files. By default, MD5 verification is disabled. If needed, it can be enabled using the –verify parameter.
To encrypt the UDP transfer, SSL must be enabled on the FileCatalyst server. The default port for encrypted FTP is usually port 990. Once the server has been configured, downloading a file requires adding –ssl and –port:
A log showing both the client-server conversations as well as file transfer statistics can be created with –summary. This can also be useful to debug situations where the login process or initial file transfer isn't working as expected.
Here is an example of how to download a file from the command-line using fccl:
When using C++, the exact same download can be accomplished using this code:
Here is the equivalent command to upload a file:
The C++ code is similar to the previous example, except for the very last line:
Additional parameters available to fccl are listed in FC++ API (core), or by running "fccl --help".
1.8.17