diff --git a/ReadMe.md b/ReadMe.md index c6cea76..cd96bf3 100644 --- a/ReadMe.md +++ b/ReadMe.md @@ -2,236 +2,25 @@ ## Summary -These scripts are to help me with various repetetive or easy-to-forget tasks -related to [Unreal Engine 4](https://www.unrealengine.com). +These scripts are to help with various tasks in [Unreal Engine 4](https://www.unrealengine.com). +They're written on the basis of using Git / Git LFS rather than Perforce (many of UE4's own +automation tools assume P4, which can be inconvenient). + +* [Setting up a project for Git / Git-LFS](./doc/GitSetup.md): including LFS locking +* [Packaging a Game](./doc/Package.md): building, cooking, archiving +* [Releasing a Game](./doc/Release.md): e.g. uploading to Itch, Steam +* [Rebuilding Lightmaps](./doc/RebuildLightmaps.md): build lighting on the command line easily (supporting git-lfs locking, no P4 dependency like RunUAT) +* [Getting Latest for Artists](./doc/GetLatest.md): pulls from git and builds so C++ changes are automatically updated +* [Cleaning Up](./doc/Cleanup.md): Deleting unneeded Hot Reload DLLs etc -* [Setting up a project for Git / Git-LFS](#git-setup) -* [Packaging Builds](#packaging-builds) -* [Releasing builds to Itch, Steam](#releasing-builds) ## Prerequisites -* Powershell 5.1 or later (preferably Powershell Core 7+) +* Powershell Core 7+ + * Almost everything is compatible with Win10 built-in PS 5.1 but 7 is better, and platform independent * PsIni module installed (library for reading INI files easily) * Run `Install-Module PsIni` in a Powershell console -## Git Setup - -Initially we'd decided to go back to Subversion because of the importance of -good locking workflow for uasset/umap files in UE4, to prevent binary merge errors. -However thanks to the good LFS and locking support in the [Git LFS 2 (beta) -plugin](https://github.com/SRombauts/UE4GitPlugin), and a couple of config tweaks, -we managed to move back to Git + LFS. - -We use a particular content workflow: - -1. All content creation files in `$REPO/MediaSrc` (subfolders by type) - * These are typically in formats e.g. Blender that UE4 doesn't read directly, so outside `Content` - * These files are *tracked* in source control -1. When exporting, output (`FBX`, `PNG`, `WAV` etc) goes in `$REPO/Content` (and subfolders) - * These files are *ignored* in source control because they are derived data - * UE4 imports them to a `.uasset` which contains all their contents anyway -1. All `.uasset` post-imported content in `$REPO/Content` is *tracked* in SVN (binary) -1. Binary content is marked as needing *locking* so UE4 will prompt you to check it out - * This is to avoid accidental parallel changes to the (many) binary files UE4 uses - * This is our overriding reason for using Subversion and not Git - -Together the scripts below configure everything so I don't have to remember. - -### Steps to set up Git + LFS for a UE4 project - -1. The script works for projects with no git repo yet, or those with an existing git repo -1. For existing repos, ideally you will not have committed any large files to Git yet - * If you have, it is *highly recommended* you use `git lfs migrate` to re-write your repository history to be LFS compatible - * `git lfs migrate import --everything --include="*.uasset,*.umap,"` - * This repo will need to be re-cloned by everyone but it's MUCH cleaner than changing to LFS mid-history -1. Make a note of any custom .gitignore entries you have, the script will replace it -1. Run `ue4-git-setup.ps1` in the root project folder -1. Add back any specialised .gitignores we didn't cover (might not need any) -1. Push ALL BRANCHES of this new repo to the host of your choice - - -## Packaging builds - -.. TODO - -## Releasing Builds - -.. TODO - - -# LEGACY: Subversion Information (ignore below if using Git) - -## Steps to create a new SVN repo for a UE4 project - -1. Create new repo on your Subversion server, note the URL e.g. https://foo/bar -1. Run `ue4-svn-create.ps1 [URL]` in the parent folder you want your project to live -1. This will result in a subfolder called "bar" by default due to URL suffix - > See [creation script usage](#subversion-repo-creation-script) to specify a different destination path -1. After it's finished the script will have created the trunk/branches/tags folders for you, - and then checked out the trunk for you into "bar" -1. You then need to copy the contents of your UE4 project into this folder - > (UE4 doesn't let you create projects in non-empty folders so if you haven't - created a project yet, create it somewhere else then move the contents) - -## Updating an existing UE4 project - -If you already have a UE4 project in Subversion, you can instead just run the -[in-place setup script](#subversion-repo-setup-script) instead to configure it -for the workflow discussed above. You'll just need to commit the changes -after the script has completed. - -## What these scripts do - -### Creation Script - -1. **Creates trunk / branches / tags folders** - - If they're missing, the script creates them and commits to the root. - -1. **Checks out trunk** - - This will become your working folder. - -1. **Runs the setup script** - - Performs all the steps from the [setup script](#-setup-script) below. - It then commits all these changes so the working copy is ready to use. - -### Setup Script - -1. **Ignore unnecessary folders** - - It ignores folders in the root we don't need in Subversion: - - * .vs - * Binaries - * Build - * DerivedDataCache - * Intermediate - * Saved - - It sets the `svn:ignore` property in the root folder to do this. It will merge - with the contents of any existing property by default. `svn:ignore` is - not set recursively and does not get inherited (despite what TortoiseSVN - gives the impression of). - -1. **Sets binary files to needs-lock** - - By setting the `svn:auto-props` value for common binary types tracked in the - repo such as `.uasset` and `.umap`, it enforces locking on those files, - triggering UE4 to give you check out prompts when you edit them. - - This property is set on the root folder and is inherited throughout the tree. - -1. **Creates folder structure** - - Subversion, and particularly TortoiseSVN has a "quirk" whereby inherited - properties are not respected for files inside a folder that isn't added to - SVN yet. This means that if you rely on `svn:global-ignores` being inherited - from a parent, a file inside a new folder that *should* be ignored is still - shown in the Add/Commit TSVN dialog, until its parent folder is added, at - which point it is hidden. - - I maintain this is *extremely* dumb behaviour because you can easily add - files that should be ignored by accident, unless you manually add the folder - first. However a big TSVN thread from a few years back has defended this - bizarre approach as "by design". So to mitigate this, I pre-create the - majority of our preferred folder structure ahead of time. - - Our preferred **workflow** is: - - 1. All content creation tool files in `$REPO/MediaSrc` (subfolders by type) - * These are typically in formats that UE4 might not read directly, so outside `Content` - * These files are *tracked* in SVN - 1. When exporting, place output (`FBX`, `PNG`, `WAV` etc) in `$REPO/Content` (and subfolders) - * These files are *ignored* in SVN because they are derived data - * UE4 imports them to a `.uasset` which contains all their contents anyway - 1. All `.uasset` post-imported content in `$REPO/Content` is *tracked* in SVN (binary) - - So the tool pre-creates `MediaSrc`, `Content` and subfolders (if they don't already - exist), and adds them to Subversion ahead of time to head off as many quirks - as possible. - -1. **Ignore derived export data in Content folder** - - As discussed in the workflow section above, exported files like `PNG` and `FBX` - are not committed because they're duplicated data (source is in Maya/Blender/Photoshop files in `MediaSrc`, game-ready data is in `.uasset` files in `Content`). So we want - to ignore them underneath `Content`. - - The `svn:ignore` property isn't helpful because it isn't inherited by subfolders, - and if you set it recursively it only applies to folders that already exist. - Although we've tried to help by creating most of the structure up-front, we - don't want to preclude creating new folders, and don't want everyone who - does so to have to remember to set the `svn:ignore` up. - - So instead we use `svn:global-ignores`, a newer property that is inherited - by children (although as noted above it behaves a little oddly before you add - folders to SVN, but at least you don't have to set the property again). Any - `PNG`, `FBX` files you export into `Content` for import will be ignored, and - you can just commit the `.uasset` files, and the original sources in `MediaSrc`. - The export files can be considered temporary. - - - > We set `svn:global-ignores` for all the common intermediate formats *only* - > on the `Content` folder. This is so that if, for some reason, you don't have - > a typical content creator file for an asset and for some reason you want to - > store the `PNG` or `FBX` for it manually (instead of just the `.uasset`, which is all - > the game needs), you can put those files in `MediaSrc` if you want, and copy - > them to `Content` for importing as a substitute for the normal export workflow. - > - > In practice I don't think you need to do this, because you can export `.uasset` - > files anyway. Probably better to just put orphaned files like this directly in - > `Content` and only commit the post-import `.uasset`. But the option is there. - - - -## Script Reference - -### Repo Creation Script - -This constructs the entirety of the starting repository from a Subversion URL -(which should contain an empty repository, or at most empty trunk/branches/tags -subfolders). - -Usage: -``` - ue4-svn-create.ps1 [-urn:]svnurl [[-path:]checkoutpath] [Options] - - -url : Subversion URL; the ROOT path (should be empty) - -path : Checkout path; if omitted append last part of URL to current dir - -help : Print this help -``` - -It calls the [setup script below](#subversion-repo-setup-script), -plus creating the trunk/branches/tags structure, and checking out trunk for you. -It also commits all the changes so your repo is ready to go. - -Unreal doesn't let you create a new project inside a directory with contents, -so create your new project elsewhere then copy it into this new trunk checkout. -Unfortunately there's no way to avoid this because we need folders to exist -to set SVN properties on them. - -### Repo Setup Script - -This is a second-level script to intialise the structure of an existing Subversion -trunk checkout for UE4. Run in the root of that trunk checkout - it will work -for existing UE4 folders or blank repositories. - -None of the changes are committed so you can review them. - -Usage: -``` - ue4-svn-setup.ps1 [[-src:]sourcefolder] [Options] - -src : Source folder (current folder if omitted) - : (should be root of trunk in new repo) - -skipstructurecheck - : Skip the check that makes sure you're in trunk - -overwriteprops - : Replace all properties instead of merging - : Will overwrite svn:ignore, svn:global-ignores, svn:auto-props - -dryrun : Don't perform any actual actions, just report on what you would do - -help : Print this help -``` +If these tools are useful, please consider :heart: **[Supporting my work on Patreon!](https://www.patreon.com/stevestreeting)** diff --git a/doc/Cleanup.md b/doc/Cleanup.md new file mode 100644 index 0000000..7870119 --- /dev/null +++ b/doc/Cleanup.md @@ -0,0 +1,3 @@ +# Cleanup tool + +TODO \ No newline at end of file diff --git a/doc/GetLatest.md b/doc/GetLatest.md new file mode 100644 index 0000000..6bbbe9a --- /dev/null +++ b/doc/GetLatest.md @@ -0,0 +1,3 @@ +# Getting Latest for Artists + +TODO \ No newline at end of file diff --git a/doc/GitSetup.md b/doc/GitSetup.md new file mode 100644 index 0000000..35eabec --- /dev/null +++ b/doc/GitSetup.md @@ -0,0 +1,57 @@ +# Git Setup + +We use Git LFS with Unreal, including [Git LFS Locking](https://github.com/git-lfs/git-lfs/wiki/File-Locking) which is now supported by +most Git hosts, including self-hosted options like [Gitea](https://gitea.io/). Locking +is important because Unreal uses a lot of binary `.uasset` files which are not mergeable. + +To make best use of git, LFS and locking, you really want to be using the +[Git LFS 2 plugin](https://github.com/SRombauts/UE4GitPlugin) for the Unreal Editor. I +also have a [fork of this plugin](https://github.com/sinbad/UE4GitPlugin) with some +improvements which haven't been merged yet. I'm still making improvements so +you might want to keep an eye on that. + +We use a this content workflow in our UE game repositories: + +1. All content creation files in `$REPO/MediaSrc` (subfolders by type) + * These are typically in formats e.g. Blender that UE4 doesn't read directly, so outside `Content` + * These files are added to Git + * They are also tracked as Git LFS files + * They are NOT marked as lockable, simply because the tooling for managing locking + isn't very good outside of Unreal right now +1. When exporting, output (`FBX`, `PNG`, `WAV` etc) goes in `$REPO/Content` (and subfolders) + * These files are *ignored* in Git because they are derived data + * UE4 imports them to a `.uasset` which contains all their contents anyway +1. Imported content becomes `.uasset` in `$REPO/Content` + * These files are added to Git + * They are also tracked as Git LFS files + * These are also marked as *lockable* in Git LFS + +Together the script below configures all of this automatically. + +## The script + +You run the script from a Powershell prompt, in the root of your UE4 project. + +``` +ue4-git-setup.ps1 [[-src:]sourcefolder] [Options] + +-src : Source folder (current folder if omitted) + : (should be root of your UE4 project) +-dryrun : Don't perform any actual actions, just report on what you would do +-help : Print this help +``` + +See the notes below for some practical details of running it. + +## Notes + +1. The script works for projects with no git repo yet, or those with an existing git repo +1. For existing repos, it's better if you have not committed any large files to Git yet + * If you have, it is *highly recommended* you use `git lfs migrate` to re-write your repository history to be LFS compatible + * `git lfs migrate import --everything --include="*.uasset,*.umap,"` + * This repo will need to be re-cloned by everyone but it's MUCH cleaner than changing to LFS mid-history +1. Make a note of any custom .gitignore entries you have, the script will replace it +1. Run `ue4-git-setup.ps1` in the root project folder +1. Add back any specialised .gitignores we didn't cover (might not need any) +1. Push ALL BRANCHES of this new repo to the host of your choice + diff --git a/doc/Package.md b/doc/Package.md new file mode 100644 index 0000000..2e1b249 --- /dev/null +++ b/doc/Package.md @@ -0,0 +1,3 @@ +# Packaging Scripts + +TODO \ No newline at end of file diff --git a/doc/RebuildLightmaps.md b/doc/RebuildLightmaps.md new file mode 100644 index 0000000..66572bb --- /dev/null +++ b/doc/RebuildLightmaps.md @@ -0,0 +1,3 @@ +# Rebuilding Lightmaps + +TODO \ No newline at end of file diff --git a/doc/Release.md b/doc/Release.md new file mode 100644 index 0000000..7f7fe9e --- /dev/null +++ b/doc/Release.md @@ -0,0 +1,3 @@ +# Release Scripts + +TODO \ No newline at end of file diff --git a/legacy/legacy-svn.md b/legacy/legacy-svn.md new file mode 100644 index 0000000..8f1cbff --- /dev/null +++ b/legacy/legacy-svn.md @@ -0,0 +1,174 @@ +# LEGACY: Subversion Information (ignore below if using Git) + +## Steps to create a new SVN repo for a UE4 project + +1. Create new repo on your Subversion server, note the URL e.g. https://foo/bar +1. Run `ue4-svn-create.ps1 [URL]` in the parent folder you want your project to live +1. This will result in a subfolder called "bar" by default due to URL suffix + > See [creation script usage](#subversion-repo-creation-script) to specify a different destination path +1. After it's finished the script will have created the trunk/branches/tags folders for you, + and then checked out the trunk for you into "bar" +1. You then need to copy the contents of your UE4 project into this folder + > (UE4 doesn't let you create projects in non-empty folders so if you haven't + created a project yet, create it somewhere else then move the contents) + +## Updating an existing UE4 project + +If you already have a UE4 project in Subversion, you can instead just run the +[in-place setup script](#subversion-repo-setup-script) instead to configure it +for the workflow discussed above. You'll just need to commit the changes +after the script has completed. + +## What these scripts do + +### Creation Script + +1. **Creates trunk / branches / tags folders** + + If they're missing, the script creates them and commits to the root. + +1. **Checks out trunk** + + This will become your working folder. + +1. **Runs the setup script** + + Performs all the steps from the [setup script](#-setup-script) below. + It then commits all these changes so the working copy is ready to use. + +### Setup Script + +1. **Ignore unnecessary folders** + + It ignores folders in the root we don't need in Subversion: + + * .vs + * Binaries + * Build + * DerivedDataCache + * Intermediate + * Saved + + It sets the `svn:ignore` property in the root folder to do this. It will merge + with the contents of any existing property by default. `svn:ignore` is + not set recursively and does not get inherited (despite what TortoiseSVN + gives the impression of). + +1. **Sets binary files to needs-lock** + + By setting the `svn:auto-props` value for common binary types tracked in the + repo such as `.uasset` and `.umap`, it enforces locking on those files, + triggering UE4 to give you check out prompts when you edit them. + + This property is set on the root folder and is inherited throughout the tree. + +1. **Creates folder structure** + + Subversion, and particularly TortoiseSVN has a "quirk" whereby inherited + properties are not respected for files inside a folder that isn't added to + SVN yet. This means that if you rely on `svn:global-ignores` being inherited + from a parent, a file inside a new folder that *should* be ignored is still + shown in the Add/Commit TSVN dialog, until its parent folder is added, at + which point it is hidden. + + I maintain this is *extremely* dumb behaviour because you can easily add + files that should be ignored by accident, unless you manually add the folder + first. However a big TSVN thread from a few years back has defended this + bizarre approach as "by design". So to mitigate this, I pre-create the + majority of our preferred folder structure ahead of time. + + Our preferred **workflow** is: + + 1. All content creation tool files in `$REPO/MediaSrc` (subfolders by type) + * These are typically in formats that UE4 might not read directly, so outside `Content` + * These files are *tracked* in SVN + 1. When exporting, place output (`FBX`, `PNG`, `WAV` etc) in `$REPO/Content` (and subfolders) + * These files are *ignored* in SVN because they are derived data + * UE4 imports them to a `.uasset` which contains all their contents anyway + 1. All `.uasset` post-imported content in `$REPO/Content` is *tracked* in SVN (binary) + + So the tool pre-creates `MediaSrc`, `Content` and subfolders (if they don't already + exist), and adds them to Subversion ahead of time to head off as many quirks + as possible. + +1. **Ignore derived export data in Content folder** + + As discussed in the workflow section above, exported files like `PNG` and `FBX` + are not committed because they're duplicated data (source is in Maya/Blender/Photoshop files in `MediaSrc`, game-ready data is in `.uasset` files in `Content`). So we want + to ignore them underneath `Content`. + + The `svn:ignore` property isn't helpful because it isn't inherited by subfolders, + and if you set it recursively it only applies to folders that already exist. + Although we've tried to help by creating most of the structure up-front, we + don't want to preclude creating new folders, and don't want everyone who + does so to have to remember to set the `svn:ignore` up. + + So instead we use `svn:global-ignores`, a newer property that is inherited + by children (although as noted above it behaves a little oddly before you add + folders to SVN, but at least you don't have to set the property again). Any + `PNG`, `FBX` files you export into `Content` for import will be ignored, and + you can just commit the `.uasset` files, and the original sources in `MediaSrc`. + The export files can be considered temporary. + + + > We set `svn:global-ignores` for all the common intermediate formats *only* + > on the `Content` folder. This is so that if, for some reason, you don't have + > a typical content creator file for an asset and for some reason you want to + > store the `PNG` or `FBX` for it manually (instead of just the `.uasset`, which is all + > the game needs), you can put those files in `MediaSrc` if you want, and copy + > them to `Content` for importing as a substitute for the normal export workflow. + > + > In practice I don't think you need to do this, because you can export `.uasset` + > files anyway. Probably better to just put orphaned files like this directly in + > `Content` and only commit the post-import `.uasset`. But the option is there. + + + +## Script Reference + +### Repo Creation Script + +This constructs the entirety of the starting repository from a Subversion URL +(which should contain an empty repository, or at most empty trunk/branches/tags +subfolders). + +Usage: +``` + ue4-svn-create.ps1 [-urn:]svnurl [[-path:]checkoutpath] [Options] + + -url : Subversion URL; the ROOT path (should be empty) + -path : Checkout path; if omitted append last part of URL to current dir + -help : Print this help +``` + +It calls the [setup script below](#subversion-repo-setup-script), +plus creating the trunk/branches/tags structure, and checking out trunk for you. +It also commits all the changes so your repo is ready to go. + +Unreal doesn't let you create a new project inside a directory with contents, +so create your new project elsewhere then copy it into this new trunk checkout. +Unfortunately there's no way to avoid this because we need folders to exist +to set SVN properties on them. + +### Repo Setup Script + +This is a second-level script to intialise the structure of an existing Subversion +trunk checkout for UE4. Run in the root of that trunk checkout - it will work +for existing UE4 folders or blank repositories. + +None of the changes are committed so you can review them. + +Usage: +``` + ue4-svn-setup.ps1 [[-src:]sourcefolder] [Options] + -src : Source folder (current folder if omitted) + : (should be root of trunk in new repo) + -skipstructurecheck + : Skip the check that makes sure you're in trunk + -overwriteprops + : Replace all properties instead of merging + : Will overwrite svn:ignore, svn:global-ignores, svn:auto-props + -dryrun : Don't perform any actual actions, just report on what you would do + -help : Print this help +``` + diff --git a/ue4-svn-create.ps1 b/legacy/ue4-svn-create.ps1 similarity index 100% rename from ue4-svn-create.ps1 rename to legacy/ue4-svn-create.ps1 diff --git a/ue4-svn-setup.ps1 b/legacy/ue4-svn-setup.ps1 similarity index 100% rename from ue4-svn-setup.ps1 rename to legacy/ue4-svn-setup.ps1