Installing a Linux system on the same media where Windows is

Ugh, another guide about installing Ubuntu from you? Stop it!

Yeah, I wrote quite a bit about that in the past, But I figured I would warm my fingers up for another long-winded typing session for people who don't want to put a Linux system on a spare media device that's half clapped-out, and would rather put their Linux installation on the same half clapped-out media device Windows exists on. That, and I've learned some new stuff since, and had dealt with other people's machines.

Before you proceed...

  • This guide assumes you don't have to mess with EFI / SecureBoot stuff
    • If you have to, that is on you.
  • If using an SSD, you still need to provision some disk space for wear leveling
    • Failure to do so, and subsequent expedited device failure is your fault
  • You follow this guide at your own risk. If you mess up, that's your dumb, not mine.
    • Provision a means to roll back, however you may. For brevity I won't mention any of that.
  • A lot of this stuff is abstract, and requires you to be in full control, with a complete understanding of your filesystem
    • If this makes you dizzy at any time, feel free to stop reading and consult a physician. Or your local IT nerd. Whatever works.
  • This guide will have some specifics about Ubuntu and Ubuntu MATE.
    • Adapt for whatever system you're using.
  • If you are using an encrypted / compressed NTFS partition, some things will become more difficult.
    • That isn't to say some things will be impossible, but more steps will be required to purpose an NTFS partition for cross-platform support.

This guide will also contain a lot of op-ed writing, so there are some opinions presented but there's nothing in here which isn't backed with and established by anything factual. This is intended for newbies, and people unfamiliar with partitioning, and it should be read top-down since a lot of what comes after makes less sense without reading what comes before.

Filesystem concepts and provisioning for dual-boot installation

Partitioning needs

If you're only interested in a single-partition installation, that is fine; you can run a system without swap of any kind and keep everything in / but that's a little foolish for several reasons;

  • If you are not vigilant about linking directories, then you'll run out of space and have to move things between partitions before updating the system
  • Lack of swap space or swapfile may cause instability on systems with less RAM available
  • If you ever have to destroy the partition for any reason, you may end up losing files that are exclusive to the Debian userland left outside of linked directories into the NTFS partition, and lose special configuration files for reinstallation (should you want to reinstall / upgrade the system).

Bear all of that in mind; partition in whatever way will inconvenience you and compromise stability the least.

Notes about encrypted / compressed NTFS partitions

Since open-source systems with NTFS support work best (i.e. work at all) with NTFS partitions that have nothing special, instead of making room for a new system outside of Windows, you'll want to consider opening diskmgmt.msc in Windows and using its tools to shrink the partition Windows is on (since you can shrink the present bootable partition in real-time), then create a new plain NTFS partition for cross-system storage later.

Note that for the purposes of exporting a lot of files it is better to move all users' library contents away from %systemdrive%, to another externally-connected drive so the system partition can be shrunk down as much as possible, while provisioning for updates and additional programs. This can be done by visiting each user's library (in %userprofile%), then right-clicking on each directory to change its properties. There is a tab called Location where you can define a new path; Windows will handle everything else.

An easy way to figure how much space to allocate for a new NTFS partition is to subtract whatever space is available (in megabytes) by how much you intend for the Linux system to occupy, then divide that by two; Use that figure to add onto the space used, and shrink to that size. If you don't have a lot in there already, that should provide enough space for extra programs and updates. After, with the media used for initial migration still connected change locations once again to the new NTFS partition.

Keeping the filesystem DRY

Don't Repeat Yourself; It is a concept used in programming all the time to not write the same code twice, and it can also be exercised on a hybrid NTFS5 / Ext4 disk setup. This will be explained later on, however in brief it is best in a dual-boot situation to repurpose existing directories for user profiles; this can be done very easily with ln in Linux, and mklink in Windows' command prompt as admin. Certain cross-platform programs can also use the same configuration files, albeit in different places; it should be considered, to squeeze every last byte out of a hard disk whether it is worth the effort of experimenting with using the same config files between systems.

Partitioning

This may differ depending on what the hard disk is capable of, but most 2.5" media I've ever encountered only allows for four primary partitions, or three with one extended.

However much space you need depends on whatever you are going to use the system for. Most setups I use on external devices go as follows;

  • Ext4 (/)
  • Ext4 (/home)
  • Extended
    • NTFS (Cross-system storage)
    • (Whatever else I may place here in the future)
  • swap

But most Windows setups are like this;

  • NTFS (Windows)
  • Fat32 / NTFS (Diagnostic software)
  • Fat32 (Norton GHOST image, built-in drivers, whatever else)

An Ext4 partition can be squeezed into there, but it might be better to make an extended partition instead so it's set up like this;

  • NTFS (Windows)
  • Extended
    • Ext4 (/)
    • Ext4 (/home)
    • NTFS (Non-compressed / -encrypted cross-system storage, if necessary)
    • swap
  • Fat32 / NTFS (Diagnostic software)
  • Fat32 (Whatever else)

If you cannot do swap space, there's no need for it; a swapfile can be made instead.

Spacial consideration

However much space you need is entirely dependent on tasks to be performed. Most open-source software doesn't need that much space; The system itself demands 10 gigabytes, and other software therein usually don't take gigabytes of space alone. Altogether additional software may use another 5 gigabytes. 56 gigabytes is a good all-around figure to set;

  • 32,768 megabytes for the system
  • 16,384 megabytes for /home
  • 8,192 megabytes for swap

If you need more space, you can use the NTFS partition that exist already for this, and all of that can be dealt with post-installation. Later on in this guide this concept will be explored. Also consider labeling the NTFS partition, since it will make certain procedures easier in the future.

How to boot

The bootloader is important to consider; If you can boot from SD or USB, it might be a good idea to keep a low-capacity SD card or USB device (512 megabytes, for example) to write the bootloader. Since it will only be needed when booting the system, it doesn't have to be used all the time and can be removed after booting is complete. That will also prevent boot issues with Windows updates, since the boot device is kept entirely separate.

Alternatively if you keep a USB device full of portable applications (FUSE software, PortableApps for Windows or stand-alone scripts and tools) you can also purpose that as a boot device, so long you are cognizant you can boot another system from it.

Executing provisioned changes

Committing partitions

Everything above this is information to put into consideration when making changes with gparted; do gksu gparted (or use sudo if gksu isn't an option) to tamper with your partitions, and make whatever changes you see fit which accommodate your needs. If you don't know what you're doing, everything I've presented before in this post is a good baseline to follow.

Only shrink the size of the NTFS partition to whatever size you need for the remaining partitions, then add in the partitions for the Ubuntu system as desired. Label as desired each partition as you are making it (extended and swap cannot be labelled), then commit your changes. Afterward, restart and confirm your changes did not ruin the Windows partition; if you can get to the Windows login screen, reboot into the live session once again to execute installation.

Installing the system

Ubiquity is a little weird about existing partitions, but they can still be purposed; at the installation screen, select which partitions you've committed for root (and home) and make them the same partition format they are right now (Ext4), but disallow formatting of these partitions; you just made them, you know what they are, and what they are suppose to do so there is no reason for Ubiquity to do anything more than copy files over and commit updates as necessary.

Use the information I presented prior about handling the bootloader to determine whether or not GRUB would be more at home in the same drive as Windows (overriding the Windows bootloader) or on a separate device (preserving the Windows bootloader). During the installation process, to make things even easier on yourself create users that are named exactly as they are on the Windows partition, so that any instance of $USER is system-independent. You can only make one user now, but others can be made in the system later.

The difference between ~ and $USER;

  • ~ refers to the filepath of /home/$USER; similar to %userprofile% in Windows
  • $USER refers to the name of the present active user

Making the NTFS partition usable

There is a handy utility included in Ubuntu MATE called gnome-disks; do gnome-disks and select the drive you installed the system onto. Select the NTFS partition in the media map, then click on the gears icon below to configure the drive so it mounts automatically. How it mounts is dependent upon what information about the partition exists; If you labeled the NTFS partition earlier, /mnt/LABEL is a good option, since it will mount in /mnt as the name of the partition, which will be easier to remember and easier to handle when moving content to another disk. Reboot once more to confirm your changes work.

Using the NTFS partition

To make use of the partition Windows exists on for your own personal storage in /home, symbolic links can be used. Using ln is the easiest way to do it, since you can go straight to business and have to rename nothing, but there needs to be special consideration for how many levels down each linked directory should be. There are two ways of handling this;

  • Single-level links per NTFS directory
  • Double-level links per NTFS directory

If you never plan on reinstalling the system from scratch, going right to the folders you need to share between systems is ideal but it may be better for long-term use to link the NTFS partition first, then make additional links from there.

For brevity I'll assume a link from the NTFS filesystem in /mnt named .ntfs is made in ~. If the previous advice about user creation was followed, then the below will work as provided with no changes necessary.

To make these links easily, here's a basic shell script you can save, chmod +x and use with .sh in the Bash shell to create links for the five most-used home directories, for any terminal emulator (mate-terminal or whatever x-terminal-emulator is as specified by update-alternatives);

# Replace "$USER" with specific usernames if necessary!
### Duplicate old contents (if any)
cp -R ~/Documents ~/.ntfs/Users/$USER/Documents
cp -R ~/Downloads ~/.ntfs/Users/$USER/Downloads
cp -R ~/Pictures ~/.ntfs/Users/$USER/Pictures
cp -R ~/Music ~/.ntfs/Users/$USER/Music
cp -R ~/Videos ~/.ntfs/Users/$USER/Videos
### Delete folders
rm -rf ~/Documents ~/Downloads ~/Music ~/Pictures ~/Videos
### Make symbolic links of directories to supersede
ln -s ~/.ntfs/Users/$USER/Documents ~/Documents
ln -s ~/.ntfs/Users/$USER/Downloads ~/Downloads
ln -s ~/.ntfs/Users/$USER/Music ~/Music
ln -s ~/.ntfs/Users/$USER/Pictures ~/Pictures
ln -s ~/.ntfs/Users/$USER/Videos ~/Videos

If you needed to use another partition instead because the NTFS partition Windows resides on is compressed or encrypted, then use Windows Explorer to change the location of the library file contents so they reside in the same working partition that will be in use when you boot into the other OS.

Conclusion

It's not easy to create a dual-boot setup. But after you're finished with initial configuration, it should be easy to maintain, and a great way to hop in head-first without losing any of your files. And when you decide the whole Linux thing isn't for you, everything about it can be deleted. If it happens the bootloader was written where Windows is, you might want to see my guide about ms-sys to fix Windows if the Windows installation media is unavailable; this can be done from the same media you used to install the system you're removing.

5 Likes

A / N: Damn, did that take a long time to write. Sorry if it became sloppy toward the end, I just wanted this out of my face.

I fixed up my script; I lied about it working previously because I forgot to specify a symbolic link (ln, rather than ln -s as it should be.)

I was contemplating a re-write that was for the most common installation scenario, but instead I’ll ust leave this;

  • Most OEMs have only two partitions, if this is the case make your Ext4 partititions in their place
  • Do not move the other partition which your Windows system isn’t in; that is asking for trouble. Instead just resize the NTFS partition, reducing it by 49153 megabytes. The extra megabyte is for the extended space which uses 1 MB / MiB to exist.
  • If encrypted or compressed, these are accommodations you need to perform;
  • Nagivate to %userprofile% in explorer and relocate your libraries to an externally-attached media.
  • Don’t use gparted: That is asking for more trouble, and while in the Windows session you should use diskmgmt.msc instead to resize.
  • Once you move your stuff off, make an extended partition in diskmgmt.msc unless the NTFS partition Windows exists on is the only one.
  • Even if it isn’t encrypted, making your content wholly separate from your system is a good practice anyway. In either case; out of necessity or desire to separate your stuff from theirs you should then use explorer to once again change the directory paths in %userprofile% so they refer to locations in the new NTFS partition before doing much of anything else.
  • For overall size, all you really need is 48 gigabytes, or 49152 megabytes;
  • 32768 megabytes for the system (/)
  • 16384 megabytes for your stuff (/home)
  • Do in the live session gksu gparted (or sudo if gksu is unavailable) and use the values above
  • You’re free to overwrite the bootloader in the drive your system is on, since it’s easy enough to fix in a live session (hopefully) and any major updates to Windows 10 shouldn’t overwrite your MBR.

Hopefully the above is enough information which is laser-focused to what would be the most common dual-boot setup. There was a slight oddity when I was following my own advice; if in an extended partition there are only partition formats that Windows do not support, they all appear as generic “Healthy” partitions Windows can’t mount, without any indication they are in an extended partition. Something to bear in mind when using diskmgmt.msc for Windows in the future.

Thanks for the tutorial! I believe it’s what I needed. A question though, if I may:

Currently, I have Windows 10 on its own SSD, UM on a second SSD, and a third disk for “Data” (3TB spinning drive) that the two share.

In Windows, I used the folder properties to physically change the location of all the user folders to the “Data” disk. In UM, I simply used the folders on the “Data” disk for downloads and saves, leaving the Home directory folders alone. UM automounts the drive at boot.

I keep having issues, however, with Windows telling me the “Data” drive needs checking, and in UM the drive keeps going Read Only. I don’t know enough about file systems to know exactly why, but assume its because the two OSs are conflicting over ownership of the disk? (I guess I should add here, that the drive is healthy, and I only have these issues when booting into one of the OSs after having written to the disk with the other)

I plan to reinstall windows soon, so was looking a a better way to handle the “Data” drive. Based on the tutorial here, I think my “Data” drive setup will be more stable if I:

  • Retain the format as NTFS
  • Windows User Folders linked, rather than moved, to the “Data” drive. Perhaps turn indexing off for the disk?
  • UM folders linked to the “Data” drive, with the drive continuing to be automounted.

Any thoughts or recommendations (from anyone) on how to do it differently/better? Thanks again.

I don’t understand why Windows needs to disk check nor why the data drive keeps going RO; maybe it’s something to do with the mount options in /etc/fstab?

gnome-disks can let you adjust how /etc/fstab handles disk mounting through a graphical interface; first figure out the RO nonsense (it might be you made it compressed by accident; the setup I specified works best with non-compressed, non-encrypted partitions), then you can specify for the disk your data stuff is on to be mounted at-boot.

Windows checking is infrequent, but the RO business is with every boot. Not compressed or encrypted. I’ve tried unmounting and remounting RW from terminal and that works, but is non-persistent. Somehwere along the way, it stopped mounting as mnt/Data and is now mounted by the UUID.

fstab looks to be set for RW, but perhaps the syntax is wrong? /dev/disk/by-uuid/202C96FF2C96CF62 /mnt/202C96FF2C96CF62 auto rw,nosuid,nodev,nofail,x-gvfs-show 0 0

My first guess now is that I created the issue tinkering with it at some point!

I don’t know. It might also help to check the flags in gparted to make sure there isn’t something that is affecting the usability of the device that way. (Let’s be honest it shouldn’t anyway but sometimes we have to check for idiocy regardless of which is at fault.)

Gparted looked like it should, but your suggestion got me thinking about another possibility which turned out to be the issue. Windows can perform a “Hybrid” (a.k.a fast) shutdown when its set in the power settings. Hybrid shuts the User and Hardware sessions down completely, but hibernates the Kernel session. Turns out I’d set it to Hybrid somewhere along the way, and the hibernated Windows Kernel apparently kept control of the NTFS disk. Setting Windows power settings for a “Complete” shutdown fixed it.

Since I only use Windows for a couple programs anymore, Hybrid shutdown keeping the Windows Kernel hibernating for 95% of the time isn’t a good idea regardless!

Thanks again for the Tutorial and recommendations! I’m plan to reinstall both windows and UM when 17.10 comes out and I think its going to be a great roadmap for what I need to do.

2 Likes

With pleasure. If you can, find yourself another Windows PC you have control of, or install Windows 10 on another machine to use temporarily so you can use Disk2VHD from Sysinternals so you can convert your present Windows installation to a virtual hard drive, if you have zero intention of physically booting into it again. There’s a guide about how to do Linux in a VM you can kind of repurpose for Windows. Some trial and error later, and you might find yourself with some extra storage space if you can retrofit your install into a virtual drive image to mount and use in Linux.

Your Windows install in a VM might benefit with a shared NTFS partition; you can dump stuff into there and it would show up in WIndows, if you can’t get drag-and-drop working in whatever solution you choose.

1 Like

How do I know if I'm in EFI messland and need to steer clear? Targeting an HP EliteBook which has win10 on it currently.

If there is an EFI partition on the disk, you're dealing with a GPT table. With my very lmited experience using one machine which was somebody else's, interacting with it remotely, you can put the bootloader into the EFI partition (in SDA) and have it work just fine.

If you are super paranoid about future updates in Windows breaking your GRUB install, simply use grub-install to put it on a USB stick and attempt to boot from it. If you get a menu, you're good. Else, specify another root device as bootloader and keep it attached, while specifying in your UEFI to always boot from USB first and you should be good to go without touching the Windows boot partition.

In fact, that's how I've been doing it. Using an old low-capacity USB stick, you can assign the bootloader to be on that root device and keep booting from it until the device wears out, which would probably be many years from now.