From 3890cf2051681cc6b6c373604973a7eb2ef4549f Mon Sep 17 00:00:00 2001 From: Steve Streeting Date: Fri, 14 May 2021 13:19:23 +0100 Subject: [PATCH] Add docs for ue4-datasync.ps1 --- ReadMe.md | 2 + doc/DataSync.md | 148 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 150 insertions(+) create mode 100644 doc/DataSync.md diff --git a/ReadMe.md b/ReadMe.md index e9f54b0..fdb9852 100644 --- a/ReadMe.md +++ b/ReadMe.md @@ -7,10 +7,12 @@ They're written on the basis of using Git / Git LFS rather than Perforce (many o automation tools assume P4, which can be inconvenient). * [Setting up a project for Git / Git-LFS](./doc/GitSetup.md): including LFS locking +* [Managing Git LFS Locking Tasks](./GitScripts/Readme.md): push and unlock, release locks you don't need any more * [Packaging a Game](./doc/Package.md): building, cooking, archiving * [Releasing a Game](./doc/Release.md): e.g. uploading to Itch, Steam * [Rebuilding Lightmaps](./doc/RebuildLightmaps.md): build lighting on the command line easily (supporting git-lfs locking, no P4 dependency like RunUAT) * [Getting Latest for Artists](./doc/GetLatest.md): pulls from git and builds so C++ changes are automatically updated +* [Synchronising BuiltData Files outside of Git](./doc/DataSync.md) * [Cleaning Up](./doc/Cleanup.md): Deleting unneeded Hot Reload DLLs etc diff --git a/doc/DataSync.md b/doc/DataSync.md new file mode 100644 index 0000000..13da5c1 --- /dev/null +++ b/doc/DataSync.md @@ -0,0 +1,148 @@ +# Synchronising "BuiltData" Files Outside Git + +## Why? + +Unreal stores lighting and other built data for your maps in a separate file. +If your level is called "YourMap.umap", then lighting data will be saved in +"YourMap_BuiltData.uasset". + +This data is entirely derived from the .umap file, and is also very large. Given +this, it's a prime candidate for exclusion from version control; there's no +need to fill up your repository with large data that can be rebuilt from the data +already tracked in it. + +However, there's a problem with doing this: + +1. Anyone else on the team won't see lighting you've built +1. Even if they take the time to rebuild lighting themselves, this process wants + to alter the base .umap file as well, since there are cross-references + +Most of the time you can get away with not including the BuiltData in the repository, +each person building lighting locally and deliberately NOT saving their map changes +when prompted. However, building the lighting gets more time consuming over time, +and remembering not to save is a pain. + +## Isn't there a standard solution? + +The solution normally presented is to use Perforce, where you can tell Perforce +to only keep the latest version of a given pattern of file. In that scenario +BuiltData files are tracked in the repository and versions other than the latest +are just deleted. + +Vanilla Git can't do this. Git LFS can, but only if you write some pruning code +specific to your LFS server. It's not ideal. + +## My solution + +My solution is to write a tool to make it easier to share BuiltData files +as a side-channel to the Git LFS repository. You simply provide a file share +(ideally a network drive, or a synced folder like Google Drive / Dropbox if you +don't mind a little duplication) and use my script `ue4-datasync.ps1` to +sync lighting data between team members. + +Using Git LFS is a prerequisite, because it uses the OIDs from the .umap files +(which must be tracked in Git LFS) as a corresponding identifier to make sure +the matching version of the BuiltData is used. + +## Details + +Note: this script will automatically close the UE4 editor if you have the +project open, in order to prevent accidental issues such as unsaved changes or +locked files. + +``` +Usage: + ue4-datasync.ps1 [-mode:] [[-path:]syncpath] [Options] + + -mode : Whether to push or pull the built data from your filesystem + -root : Root folder to sync files to/from. Project name will be appended to this path. + : Can be blank if specified in UE4SYNCROOT + -src : Source folder (current folder if omitted) + : (should be root of project) + -prune : Clean up versions of the data older than the latest + -force : Copy ALL BuiltData files regardless of size/timestamp checks + -nocloseeditor : Don't close UE4 editor (this will prevent download of updated files) + -dryrun : Don't perform any actual actions, just report on what you would do + -verbose : Print more information + -help : Print this help + +Environment Variables: + UE4SYNCROOT : Root path to sync data. Subfolders for each project name. + UE4INSTALL : Use a specific UE4 install. + : Default is to find one based on project version, under UE4ROOT + UE4ROOT : Parent folder of all binary UE4 installs (detects version). + : Default C:\Program Files\Epic Games +``` + +You must tell the sync tool where the shared drive is, either using the `-root` +argument, or defining the `UE4SYNCROOT` environment variable. A project folder +will be added below that, based on the name of your .uproject file, so that +you can use the same root folder for multiple projects. + +The tools works in "push" or "pull" mode, and processes all .umap files which +are tracked in Git LFS (others are ignored). It's worth explaining exactly +what happens in each mode. + +### Push mode + +> Example: `ue4-datasync.ps1 push` +> +> Assuming you run this in your project root and have defined the environment variable UE4SYNCROOT + +In push mode, you want to upload BuiltData files you've updated, probably because of a +change to the .umap. You have to have committed your changes to the .umap first, +the tool won't allow you to push changes if they're uncommitted (to avoid drifting changes). + +For each umap file, the Git LFS OID (basically a SHA256 of the umap file) for your +current version is used to derive a version-specific filename, e.g. "YourMap_BuiltData_112233445567.uasset". +Your local copy of the BuiltData file will be copied to the shared drive with this +name (under the subfolder Project/ContentPath). If there's already a file named +this in the shared folder, and the size & date/timestamp match, then nothing will happen, +unless you use the `-force` argument. + +Because you're not allowed to have local uncommitted changes to the umap files, +and the BuiltData is tagged with the SHA of the umap, this means other people can +get a 'safe' copy of your current lighting build corresponding to the state of the +umap on this shared drive, without it being in the git repo. + +### Pull mode + +> Example: `ue4-datasync.ps1 pull` +> +> Assuming you run this in your project root and have defined the environment variable UE4SYNCROOT + +In pull mode, the script tries to find the BuiltData files corresponding to your +umap files on the shared drive. Again, you can't have any uncommitted changes to +umap files. + +In the same way as push, the script uses the OID of the umap file to look up +the versions on the shared drive. If the matching OID file exists, and +you don't have a newer local version, then the BuiltData file is copied into +place in your local project folder. Next time you open the editor you'll +have the lighting data that your team mate built. + +## Pruning + +By default, the different versions of BuiltData would build up on the shared +drive, one per OID of .umap file. To clean up and only keep the latest, +add the `-prune` option (this isn't enabled by default because destructive actions +should always be opt-in). + +The prune routine looks at all your tracked .umap files, and then deletes any +files on the shared drive for this umap that have OIDs *other than* the current +version, and which have an older modification date (to prevent accidental deletion +of newer versions someone else has recently pushed after the version you're on). + +Although you can provide `-prune` to any invocation of this script, I'd recommend +you only do it for the `push` variant. + +## Automating this + +You can use `ue4-datasync.ps1` manually, calling it in `push` mode just after +you push any map changes (assuming you've built the lighting), and in `pull` mode +on demand, as and when you know you want to pick up new lighting data that others +have built. + +Alternatively you could add these commands to your git hooks, perhaps `pre-push` +and `post-checkout` for `push` and `pull` respectively. Given the lower frequency +of lighting build changes though, the manual approach is probably fine. \ No newline at end of file