Dear Users! Please post bug reports about FDM here!

FDM 3.9.4.1481 + HTTPS+ NTLM + files larger that 2.0gb = 401

Moderators: Usher, Alex

knighthawk
Posts: 1
Joined: Mon Dec 01, 2014 7:14 pm

FDM 3.9.4.1481 + HTTPS+ NTLM + files larger that 2.0gb = 401

Postby knighthawk » Mon Dec 01, 2014 9:41 pm

So I've been having the following issue for years with FDM and I'd really like to see it get resolved if possible, or at least know I'm not the only one that has come across this - searching has never seemed to produce results very specific to my issue.

Issue\Problem:
Attempting to download a file via FDM 3.9.4.xxxx from an IIS webserver via https(SSL\TLS) url where the folder or files are secured by windows authentication\NTLM will fail with a 401.Unauthorized response if the file size is greater than 2gb(2,147,483,648 bytes). Attempting to download a file that is say 2,135,000,000 bytes via FDM from the same server\IIS setup will result in a successful download.

Expected outcome: The file larger than 2gb to download successfully over ssl\tls when using ntlm. ;)

Things of note:
-Grabbing the 2gb+ file in a modern browser completes successfully without issue (after authenticating of course).
-When one temporarily disables the auth for said 2gb+ file, say by setting it to anonymous the file will download just fine.
-I've reproduced this over the years on II6,II7,II7.5 and various prior iterations of FDM 3.x

-While I can't say it didn't exist in the 2.x time frame I don't recall running across it till the 3.x line, or it could just be that the number of times during the 2.x.x years I was grabbing 2gb files back then via fdm+https+ntlm was near zero.

Sample Logs (Success) 1.4gig:

Code: Select all

10:36:59 PM 11/30/2014 Starting download...
10:36:59 PM 11/30/2014 GET /test/smaller.avi HTTP/1.1
Referer: https://192.168.200.10:8443/test/
Host: 192.168.200.10
10:36:59 PM 11/30/2014 HTTP/1.1 200 OK
Content-Type: video/x-msvideo
Last-Modified: Tue, 11 Nov 2014 00:58:58 GMT
Accept-Ranges: bytes
ETag: "ac2fdac4afdcf1:0"
Server: Microsoft-IIS/7.5
Persistent-Auth: true
X-Powered-By: ASP.NET
Date: Mon, 01 Dec 2014 03:38:24 GMT
Content-Length: 1400864442
10:36:59 PM 11/30/2014 Opening file on the disk...
10:36:59 PM 11/30/2014 Succeeded
10:36:59 PM 11/30/2014 [Section 1] - Started
10:36:59 PM 11/30/2014 [Section 1] - Downloading
10:37:03 PM 11/30/2014 Preparing files on the disk... This may take several minutes
10:37:30 PM 11/30/2014 Creating new section...
10:37:30 PM 11/30/2014 GET /test/smaller.avi HTTP/1.1
Referer: https://192.168.200.10:8443/test/
Range: bytes=724982111-
Host: 192.168.200.10
10:37:31 PM 11/30/2014 HTTP/1.1 206 Partial Content
Content-Type: video/x-msvideo
Last-Modified: Tue, 11 Nov 2014 00:58:58 GMT
Accept-Ranges: bytes
ETag: "ac2fdac4afdcf1:0"
Server: Microsoft-IIS/7.5
Persistent-Auth: true
X-Powered-By: ASP.NET
Date: Mon, 01 Dec 2014 03:38:55 GMT
Content-Length: 675882331
Content-Range: bytes 724982111-1400864441/1400864442

.......etc..etc... for other sections.


Log Sample (Failure) 2.1gb

Code: Select all

9:29:04 AM 12/1/2014 Starting download...
9:29:04 AM 12/1/2014 GET /test/Larger.mkv HTTP/1.1
Referer: https://192.168.200.10:8443/test/
Host: 192.168.200.10
9:29:05 AM 12/1/2014 HTTP/1.1 401 Unauthorized
Content-Type: text/html
Server: Microsoft-IIS/7.5
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
Date: Mon, 01 Dec 2014 14:30:27 GMT
Content-Length: 1293
9:29:05 AM 12/1/2014 Opening file on the disk...
9:29:05 AM 12/1/2014 Succeeded
9:29:05 AM 12/1/2014 [Section 1] - Started
9:29:05 AM 12/1/2014 [Section 1] - Downloading
9:29:05 AM 12/1/2014 [Section 1] - Done
9:29:05 AM 12/1/2014 Download complete
10:50:51 AM 12/1/2014 Starting download...
10:50:51 AM 12/1/2014 GET /test/Larger.mkv HTTP/1.1
Referer: https://192.168.200.10:8443/test/
Host: 192.168.200.10
10:50:52 AM 12/1/2014 HTTP/1.1 401 Unauthorized
Content-Type: text/html
Server: Microsoft-IIS/7.5
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
Date: Mon, 01 Dec 2014 15:52:21 GMT
Content-Length: 1293 
10:50:52 AM 12/1/2014 Opening file on the disk...
10:50:52 AM 12/1/2014 Succeeded
10:50:52 AM 12/1/2014 [Section 1] - Started
10:50:52 AM 12/1/2014 [Section 1] - Downloading
10:50:52 AM 12/1/2014 [Section 1] - Done
10:50:52 AM 12/1/2014 Download complete

The second just produces and saves a ~2k html page with the "401 - Unauthorized: Access is denied due to invalid credentials." return message.

Additional Info:
In some reading it seems the first 2 or 3 401.unauthorized are quite normal in the process of curl ( or wget,etc) working through the username\pwd acceptance and negotiation process. I am by no means a programmer but sniffing around the FDM source a little it appeared like curl is used to do the request, then again my ability to follow cpp code is limited to say the least. The number of ntlm related fixes since 7.19.2 (ver stamp in the source) are more than a few though so I thought that might be a direction if any to point toward though, or perhaps FDM see's the first 401 and just balks? I don't know, doesn't really explain the 2gb barrier weirdness in an obvious way though unless large file support has some different codepath. Then again I really should leave the tachyon modulation to those closest too it. ;) I'll be darned if I could find 7.19.2 curl binaries to try and manual reproduce the behavior, but 7.20x64 and the current 7.39 win32 & win64 bit seemed to work fine when testing with:

Code: Select all

curl -v --anyauth -k -u FDM:fdmtest569 -O https://192.168.200.10:8443/fdm/Larger.mkv

It'll produce 2 or 3 401's during negotiation and then successfully start the dl.



Repro steps (abbreviated version):
1. Install\Setup IIS (we'll take 7.5 for example, include Dir-browsing and the various authtypes and management tools), assign a cert (self-signed or otherwise shouldn't matter), and bind it to the ip:port of your choice (192.168.200.10:8443 for my example).

2. Create a local user account, we'll call him FDM, Create some local folder (c:\fdm) and give FDM read\exec rights.

3. Pop into the IIS management MMC and create new VDir called downloads, map it's root to c:\fdm, enable directory browsing for the new Vdir, delete the default document if there is one.

4. In the ssl settings for the vdir - click "require ssl" and ignore client certs. In the authentication settings for the vdir enable windows auth, disable all other auth types. In the authorization settings for the vdir add an Allow rule for user FDM. In the request-filtering settings, on the action tab click edit feature settings and set the max content length to 4294967295 (disables it).

5. In c:\fdm create place a file say called Smaller.dat of say 1.9gb in size, and one Larger.dat that is 2.3gb in size.

6. Verify you can login (and are forced to) to the https://192.168.200.10:8443/fdm site and are presented the directory listing from a browser on another system, if not successful permissions haven't been set right most likely - go back and double check that. Verify you can grab each file via your browser (no need to dl the whole thing so long as the DL starts for a few seconds)

7. Now attempt the downloads via FDM (either manually or right click 'download with fdm' routine), including specifying that the site requires login options and the user\pass, the smaller one will work, the larger(2.1gb) will fail with 401 unauthorized.

or...
PM me and I can just provide a site to test against.

Return to “General forum”

Who is online

Users browsing this forum: No registered users and 1 guest