Plex, rclone mounts and bandwidth usage

flips · December 2020

I've been having some fun lately, setting up Plex on a new VPS, mounting my music share using rclone mount.

I've been using the same setup previously, on a slower node, but with only outbound bandwidth measured. There I mounted my music from pCloud.

On this new node I figured I would try out/use @koofr. So I mounted the drive, using

/usr/bin/rclone mount --uid 1000 --gid 1000 --syslog --stats 1m \
  -v --allow-other --read-only  myKoofr:/music/ /srv/music/

I installed plexmediaserver and added 149 GB of music to the library, from that Koofr-rclone mount.
That triggered Plex to download ~900 GB of data ...
(Actually jumped in at ~600 and tried figuring out what's going on. By then it was halvway through indexing/scanning the library. @AnthonySmith suggested I could try to add --ignore-checksum, so I did. This might have helped, as it "only" used 300 GB for scanning the last 150 GB.

I also tried pCloud, and I tried adding some more options for caching, using something like this:

/usr/bin/rclone mount --uid 1000 --gid 1000 --syslog --stats 1m \
  --buffer-size 1G --no-modtime --dir-cache-time 90m \
  --cache-dir /tmp/rclone-cache --vfs-cache-mode full \
  --vfs-cache-max-size 1G \
  -v --allow-other --read-only --ignore-checksum \
  my-pcloud:/music/ /srv/music/

Still a rescan of my library used 150 GB of traffic. Maybe --no-modtime prevents it from seeing ctime/mtime, so it has to re-download every file to check ... D'oh!

Trying again, a Scan Library Files after removing that option, with only 3 albums (re)added, it seems that still triggered a full re-download ... (Actually waiting for it to finish, but looks like it's downloading a lot.) I would think Plex stored in it's db the timestamps etc, so it wouldn't have to redownload everything every time.

I've seen recommendations on using sshfs instead, but this makes me curious ... I could of course try davs/dav2s (the cloud providers don't offer ssh, I think). But users like @Mason and @Wolveix do run rclone (albeit with GSuite, IIRC), so it should be feasible.
If nightly rescan triggers redownload of every file, it's really not very nice ... Not a vital service for me, of course, but it triggers my curiosity.

Maybe I'm missing something obvious, or maybe you have some insight to share on this?

deepak_leb · December 2020

Ok it's technical

flips · December 2020

Wasn't sure if it was mostly technical or mostly off-topic

Mason · December 2020

@flips said: do run rclone (albeit with GSuite, IIRC), so it should be feasible.

Aye, I use GSuite mounted via rclone. I agree doing it this way does generate a lot of bandwidth/traffic (mostly incoming, little outgoing except when serving content).

Some tweaks I'd suggest, you can increase your "--dir-cache-time" flag in your second mount command. I personally use 72h. Will keep things hanging around in cache longer, but also detects changes in the mount for newly added files so don't worry about it not picking up new additions.

Second, check your PMS' "Library" settings. You can disable a lot of the automatic scans so that scans aren't kicking off so frequently (see below). You can up the library scan interval to the max ("Daily") to further limit the number of re-scans.

Third, check your PMS' "Scheduled Tasks" settings. Untick the 'Refresh local metadata every three days', 'Update all libraries during maintenance', 'Refresh music library metadata periodically', and 'Perform extensive media analysis during maintenance' settings.

Closing thoughts -- If you do all the above, then scans should only occur during the daily re-scan and not when files get added/changed (or when you trigger a manual scan via Library Name (in sidebar) -> three dots -> "Scan Library Files"). Your use case does seem really high in the amount of bandwidth that it uses, which may be a side effect of koofr/pCloud. I know for GDrive, it'll download the file in chucks which (to Google) translates to the file being downloaded multiple times. And depending on the media type, the whole file might need to be downloaded in order to do media/metadata analysis on the file (I don't use Plex for music so I can't comment there). I wouldn't recommend using sshfs, it'll way underperform compared to rclone w/ vfs caching. For GSuite in particular, sshfs is way more likely to cause temporary bans due to the high number of API calls (not sure how other providers handle this).

InceptionHosting · December 2020

@Mason said: I wouldn't recommend using sshfs

Would sshfs not stop the entire download though in a bandwidth-constrained environment?

I recommended that as that was my thinking, assuming performance does not degrade too much that is.

I need to go through a similar exercise myself, I was thinking of testing sshfs first failing that, iscsi, I believe the key is to get the head to believe the data is local, which is never will with rclone etc.

Mason · December 2020

@AnthonySmith said: Would sshfs not stop the entire download though in a bandwidth-constrained environment?

My opinions are mainly based on using Plex for video streams, so it may perform quite well for smaller (i.e. music) files. So I guess take my advice with a grain of salt if you're only using Plex for music. When I used a FUSE-based mount, I had a lot of issues with the stream just taking forever to finally start, skipping didn't work well, and the API issues I mentioned due to not caching the content. Granted, I've been married to rclone for a good two or three years now, so my knowledge on this is quite outdated.

Would be interested in how an experiment goes under these various conditions.

InceptionHosting · December 2020

@Mason said:

@AnthonySmith said: Would sshfs not stop the entire download though in a bandwidth-constrained environment?

My opinions are mainly based on using Plex for video streams, so it may perform quite well for smaller (i.e. music) files. So I guess take my advice with a grain of salt if you're only using Plex for music. When I used a FUSE-based mount, I had a lot of issues with the stream just taking forever to finally start, skipping didn't work well, and the API issues I mentioned due to not caching the content. Granted, I've been married to rclone for a good two or three years now, so my knowledge on this is quite outdated.

Would be interested in how an experiment goes under these various conditions.

Same here, usually 1 - 2 GB Movies.

I will give it a test and see how it goes.

saibal · December 2020

Newer Plex versions scan your TV Shows for intros and show a button to skip opening credits. You can disable that too. The option should be buried somewhere in the preferences.

@Mason said: For GSuite in particular, sshfs is way more likely to cause temporary bans due to the high number of API calls (not sure how other providers handle this).

Can confirm this. I would get banned everyday before I started using rclone.

chocolateshirt · December 2020

My experience adding about 1500 music album, I use small vfs-read-chunk-size 8MB, 128MB for vfs-read-cache-limit and use 16MB for buffer-size for rclone Gdrive mount.

Also, I added 20 to 30 album per-scan to make sure Plex scan effective. I would avoid automatic & periodic scan and only ticked "Run partial scan when changes are detected". If you have high RAM VPS/server you could increase the amount of album per scan.

Complete scan of 280 GB flac library take around 1000 GB of bandwidth.

This is my mount config on 1GB VPS using rclone v.1.52.3

/usr/bin/rclone mount \
        --rc \
        --allow-other \
        --buffer-size 16M \
        --multi-thread-streams 0 \
        --dir-cache-time 48h \
        --tpslimit 10 \
        --vfs-read-chunk-size 8M \
        --vfs-read-chunk-size-limit 128M \
        -v \
        --async-read=false \

flips · December 2020

First of all: Thanks for great replies!

@Mason said: Closing thoughts -- If you do all the above, then scans should only occur during the daily re-scan and not when files get added/changed (or when you trigger a manual scan via Library Name (in sidebar) -> three dots -> "Scan Library Files"). Your use case does seem really high in the amount of bandwidth that it uses, which may be a side effect of koofr/pCloud. I know for GDrive, it'll download the file in chucks which (to Google) translates to the file being downloaded multiple times. And depending on the media type, the whole file might need to be downloaded in order to do media/metadata analysis on the file (I don't use Plex for music so I can't comment there).

Thanks! I'm thinking nightly scans will not be feasible if it's going to use 150 GB per night for files that doesn't change much.
Looking into @chocolateshirt's suggestions on vfs-chunk-size (was already considering those).

@chocolateshirt said: Complete scan of 280 GB flac library take around 1000 GB of bandwidth.

I wonder why it reads the files multiple times ... AirSonic and JellyFin used ~152 GB of traffic to scan ~150 GB of music ...

Trying your settings. Trigger to trigger partial scan, how to make Plex see the change ... Changed the name of a directory, didn't seem to help. I see you don't use --read-only(?), make that makes a difference.

This is my mount config on 1GB VPS using rclone v.1.52.3
/usr/bin/rclone mount \
--rc \

What does --rc do? I don't see it in the web page on rclone-mount ...
(Edit: Found it, commented below.)

InceptionHosting · December 2020

To be honest, this thread to me is looking more like an advert to have your media (whatever it is) on a local array if that is at all possible.

forces · December 2020

try the following flags instead

--allow-other \
--copy-links \
--no-gzip-encoding \
--no-check-certificate \
--allow-non-empty \
--umask 000 \
--vfs-cache-mode writes \
--no-modtime \
--max-read-ahead 100M \
--drive-chunk-size 64M \
--checkers=16 \
--dir-cache-time=160h \
--cache-info-age=60m

i mounted 3 drives with over 1tb of media
never run into any issue
i use it for jellyfin tho

flips · December 2020

@AnthonySmith said: To be honest, this thread to me is looking more like an advert to have your media (whatever it is) on a local array if that is at all possible.

You might have a point! But maybe there's vital changes in how GDrive and pCloud and Koofr handle things ... (?)

@forces said:
try the following flags instead

--allow-other \
--copy-links \
--no-gzip-encoding \
--no-check-certificate \
--allow-non-empty \
--umask 000 \
--vfs-cache-mode writes \
--no-modtime \
--max-read-ahead 100M \
--drive-chunk-size 64M \
--checkers=16 \
--dir-cache-time=160h \
--cache-info-age=60m

i mounted 3 drives with over 1tb of media
never run into any issue
i use it for jellyfin tho

Is this on Google Drive/Suite?

@chocolateshirt: Found the --rc option mentioned under Non Backend Flags:
"Enable the remote control server."
... I do wonder what this means, though.

chocolateshirt · December 2020

--rc

I added it to mount to make a remote control on it.

How "Run partial scan when changes are detected" work.

I will give you an example:

1 -- First you added several album of Avril Lavigne on Music folder, the tree will be like this:

Music
└─ Avril Lavigne
   ├─ Album A
   ├─ Album B
   └─ Album C

2 -- You scan it and plex index all the albums of Avril Lavigne. Then you added another album of Avril and other artist, so the tree will be:

Music
├─ Avril Lavigne
│  ├─ Album A
│  ├─ Album B
│  ├─ Album C
│  ├─ Album D
│  └─ Album E
└─ Justin Bieber
   ├─ Album A
   └─ Album B

3 -- On the next scan, Plex will scan only Avril Lavigne - Album D, Avril Lavigne - Album E, Justin Bieber - Album A, and Justin Bieber - Album B. It will not scan unchanged files/folders, so it will not re-read/download the existing/scanned music. You can save a lot of bandwidth with this step.

flips · December 2020

@chocolateshirt What I didn't understand is what a remote control means in this setting ...

Also, I tried adding dirs as you outline, but Plex didn't seem to pick up/be notified of the change and trigger any partial scan ...

chocolateshirt · December 2020

@flips said:
@chocolateshirt What I didn't understand is what a remote control means in this setting ...

Also, I tried adding dirs as you outline, but Plex didn't seem to pick up/be notified of the change and trigger any partial scan ...

with remote control, you could adjust some parameter without restart the mount services. About folder change, could you try to ls on the mount folder to make sure the rclone mount update the folder/files list. If there is no update then plex will not scan new file/folder, because it is not yet exist.

forces · December 2020

@flips said:

@AnthonySmith said: To be honest, this thread to me is looking more like an advert to have your media (whatever it is) on a local array if that is at all possible.

You might have a point! But maybe there's vital changes in how GDrive and pCloud and Koofr handle things ... (?)

@forces said:
try the following flags instead

--allow-other \
--copy-links \
--no-gzip-encoding \
--no-check-certificate \
--allow-non-empty \
--umask 000 \
--vfs-cache-mode writes \
--no-modtime \
--max-read-ahead 100M \
--drive-chunk-size 64M \
--checkers=16 \
--dir-cache-time=160h \
--cache-info-age=60m

i mounted 3 drives with over 1tb of media
never run into any issue
i use it for jellyfin tho

Is this on Google Drive/Suite?

@chocolateshirt: Found the --rc option mentioned under Non Backend Flags:
"Enable the remote control server."
... I do wonder what this means, though.

mounted 3 disk, one gsuite 2 onedrive
rclone will auto recgonized if the flags isnt usable and will ignore if there is any N/A flag
should work with any drive, i tested onedrive gdrive and mega
i schedule my server auto scan media 3 times a day. mount and unmount disk 3 times a day as well

Mason · December 2020

@forces said: mount and unmount disk 3 times a day as well

Just curious, why? Run into issues of some sort?

I've left my mount untouched and active for months at a time without any issues.

forces · December 2020

@Mason said:

@forces said: mount and unmount disk 3 times a day as well

Just curious, why? Run into issues of some sort?

I've left my mount untouched and active for months at a time without any issues.

too frequent upload to my drives and cant wait for it to refresh itself
i wonder should i make the cache time lower too?

Mason · December 2020

@forces said:

@Mason said:

@forces said: mount and unmount disk 3 times a day as well

Just curious, why? Run into issues of some sort?

I've left my mount untouched and active for months at a time without any issues.

too frequent upload to my drives and cant wait for it to refresh itself
i wonder should i make the cache time lower too?

Not sure about other services, but GDrive over rclone should pick up changes right away unless you're doing some aggressive caching that I'm not aware of. I often upload stuff and run a scan immediately after without missing any of the new additions.

flips · December 2020

@chocolateshirt said:

@flips said:
@chocolateshirt What I didn't understand is what a remote control means in this setting ...

Also, I tried adding dirs as you outline, but Plex didn't seem to pick up/be notified of the change and trigger any partial scan ...

with remote control, you could adjust some parameter without restart the mount services.

Oh, really, that's nice! How do you adjust parameters, then?
(So far I have stopped plexmediaserver, unmounted, fixed the mount script, remounted and restarted plexmediaserver.)

About folder change, could you try to ls on the mount folder to make sure the rclone mount update the folder/files list. If there is no update then plex will not scan new file/folder, because it is not yet exist.

Changes are visible, when I add a folder, I can see it on the VPS where it's mounted using ls etc.
But it doesn't seem like Plex is notified. (Not sure which notify mechanism it would use/try to utilize, either ...)

Anyways, no big deal, I'll just hit the button to scan when I have added stuff.

And now, it seems everything works smoothly! When I rescan now, it's really quick, just picking up if there's any changes.

My current options are:

/usr/bin/rclone mount --uid 1000 --gid 1000 --syslog --stats 1m \
  --buffer-size 16M --dir-cache-time 48h \
  --rc \
  --poll-interval 0 \
  --multi-thread-streams 0 \
  --tpslimit 10 \
  --vfs-read-chunk-size 8M \
  --vfs-read-chunk-size-limit 128M \
  --async-read=false \
  --cache-dir /tmp/rclone-cache --vfs-cache-mode full \
  --vfs-cache-max-size 1G \
  -v --allow-other --read-only --ignore-checksum \
  pCloud:/music/ /srv/music

Seems to work like a charm now, and even when I selected all albums and hit Refresh metadata, it only used ~130 GB of traffic. (The library is now 142 GB.)
Not sure which of the options did the trick, but I'm happy.
Thanks guys! (Especially @AnthonySmith, this VPS has awesome performance!)

And BTW, PlexAmp now has loop/repeat option, and the regular Plex app also has the option to shuffle and unshuffle the running queue. (Only thing I'm missing in PlexAmp now.)
Plex's scanner is still quite buggy, especially with non-ascii 7 bit characters, it seems. (But for most a refresh of metadata fixes it.)

chocolateshirt · December 2020

@flips said:

About folder change, could you try to ls on the mount folder to make sure the rclone mount update the folder/files list. If there is no update then plex will not scan new file/folder, because it is not yet exist.

Changes are visible, when I add a folder, I can see it on the VPS where it's mounted using ls etc.
But it doesn't seem like Plex is notified. (Not sure which notify mechanism it would use/try to utilize, either ...)

Anyways, no big deal, I'll just hit the button to scan when I have added stuff.

Yes, you need manual scan everytime you added new folder. I never used automatic one, hehe.. I am glad your mount working properly now.

Wolveix · December 2020

Late to this thread. Currently have 180TB in GSuite (~75000 episodes, ~9000 movies, ~106000 songs), utilizing multiple Shared Drives instead of My Drive to exceed the daily limits via Service Accounts. My average movie is around 8GB, which utilizes the recommended bitrate of 8mbps - 12mbps for a 1080P stream. You don't need to set a lot of VFS flags anymore, as Rclone has them set by default. Here's my current mount command:

rclone mount remote: /local/path --allow-other --log-level INFO --log-file /path/to/log --user-agent="PutARandomStringInHere" --uid 1001 --gid 1001 --buffer-size 1G --fast-list --dir-cache-time 1000h --poll-interval 5s --allow-non-empty --daemon

@chocolateshirt said: Yes, you need manual scan everytime you added new folder. I never used automatic one, hehe.. I am glad your mount working properly now.

Regarding media scans, if you're using Lid/Rad/Sonarr, use Autoscan, and disable all forms of automated library scans within Plex. This will send scan jobs to Plex for individual folders when a download is imported.

forces · December 2020

@Mason said:

@forces said:

@Mason said:

@forces said: mount and unmount disk 3 times a day as well

Just curious, why? Run into issues of some sort?

I've left my mount untouched and active for months at a time without any issues.

too frequent upload to my drives and cant wait for it to refresh itself
i wonder should i make the cache time lower too?

Not sure about other services, but GDrive over rclone should pick up changes right away unless you're doing some aggressive caching that I'm not aware of. I often upload stuff and run a scan immediately after without missing any of the new additions.

oh, i never know gdrive can do that! i used mainly onedrive and dont seem to pick up update and usually need a remount...

dahartigan · December 2020

@forces said:

@Mason said:

@forces said:

@Mason said:

@forces said: mount and unmount disk 3 times a day as well

Just curious, why? Run into issues of some sort?

I've left my mount untouched and active for months at a time without any issues.

too frequent upload to my drives and cant wait for it to refresh itself
i wonder should i make the cache time lower too?

Not sure about other services, but GDrive over rclone should pick up changes right away unless you're doing some aggressive caching that I'm not aware of. I often upload stuff and run a scan immediately after without missing any of the new additions.

oh, i never know gdrive can do that! i used mainly onedrive and dont seem to pick up update and usually need a remount...

@Mason any idea which rclone flags help make that work?

Mason · December 2020

@dahartigan said:

@forces said:

@Mason said:

@forces said:

@Mason said:

@forces said: mount and unmount disk 3 times a day as well

Just curious, why? Run into issues of some sort?

I've left my mount untouched and active for months at a time without any issues.

too frequent upload to my drives and cant wait for it to refresh itself
i wonder should i make the cache time lower too?

Not sure about other services, but GDrive over rclone should pick up changes right away unless you're doing some aggressive caching that I'm not aware of. I often upload stuff and run a scan immediately after without missing any of the new additions.

oh, i never know gdrive can do that! i used mainly onedrive and dont seem to pick up update and usually need a remount...

@Mason any idea which rclone flags help make that work?

Not really sure, but if I had to venture to guess, it may be the "--poll-interval" flag that @Wolveix has in his mount command above that'll do the trick.

Just make it low, like the 5s he uses

dahartigan · December 2020

@Mason said:

@dahartigan said:

@forces said:

@Mason said:

@forces said:

@Mason said:

@forces said: mount and unmount disk 3 times a day as well

Just curious, why? Run into issues of some sort?

I've left my mount untouched and active for months at a time without any issues.

too frequent upload to my drives and cant wait for it to refresh itself
i wonder should i make the cache time lower too?

Not sure about other services, but GDrive over rclone should pick up changes right away unless you're doing some aggressive caching that I'm not aware of. I often upload stuff and run a scan immediately after without missing any of the new additions.

oh, i never know gdrive can do that! i used mainly onedrive and dont seem to pick up update and usually need a remount...

@Mason any idea which rclone flags help make that work?

Not really sure, but if I had to venture to guess, it may be the "--poll-interval" flag that @Wolveix has in his mount command above that'll do the trick.

Just make it low, like the 5s he uses

Thanks for your reply, I'll try that when I get the chance and report back for science.

flips · December 2020

For cleaning up metadata, do you just fix the tags in files and rescan the library, or do you delete/move away files, rescan library, fix tags and re-add and then rescan library? (With a much slower VPS, the latter was the only option that seemingly worked.)

Mason · December 2020

@flips said:
For cleaning up metadata, do you just fix the tags in files and rescan the library, or do you delete/move away files, rescan library, fix tags and re-add and then rescan library? (With a much slower VPS, the latter was the only option that seemingly worked.)

Should be able to update the tags, then hit the 'refresh metadata' option for that song/album/artist/whatever fidelity you need (even the whole library, but that would take awhile)

flips · December 2020

I'll give it try
It failed previously, but that 512 MB vps struggled on some operations ...
(Did have plenty of slow HDD swap.)

Mew · December 2020

Oh, you running all of this on a low end 512mb vps? Whats the streaming experience for you like? I've wanted to do the same since I don't need the transcoding, but I'm not sure yet about the performance and latency on the lower end.

Plex, rclone mounts and bandwidth usage

Comments

How "Run partial scan when changes are detected" work.