Not all downloads are born equal

Over the past few years download speeds on mobile networks have been increasing to the point where I actually have better speeds on my 4G connection than my fixed boardband connection (I must live on one of the last streets in London without fibre). Once upon a time, an app was a stripped down version of a website (albeit with much better animations) but now users now expect more from their apps than ever before. One way that we are meeting these increased expectations is by producing increasingly media rich experiences. Despite the above mentioned increases in download speeds we often come up against a network bottleneck were the user has to wait on the media being downloaded. This is because downloading one image or video can be the equivalent of making multiple text-only-response requests. In order to counter this slow down we often take tight control over how we schedule and cancel those media requests. But even doing this we still end up with duplicate and/or wasted requests, which drive up loading times in our apps and leads to a poorer experience for our users. A long loading time is probably second only to a crashing bug in destroying a user's enjoyment of our apps.

Most media heavy apps require more than a simple download system that only knows how to schedule and cancel requests, we need something smarter. This smarter system needs to understand that not all media download requests are created equal - a different approach should be deployed for downloading preview images that will be displayed in a tableview than for downloading the full-size image that will be shown alone in the UI. This system also needs to understand that merely cancelling a media request could in fact be counter-productive if the user returns back to that screen.

Below we are going to look at building that download system 👩‍🔬.

Getting access to some content 📡

Before we can build this new download system we need to get some content to display - thankfully Imgur has a great JSON based API and a large library of freely available media. Imgur API, while being free, does require us to register our app with them in order to get a client-id which we will then send with each request.

In order to run the example project, you will need to register.

As the example project does not require access to a user's personal data, when registering select the Anonymous usage without user authorization option for Authorization type:

Screen shot of authorization type options

At the time of writing you had to provide a value in the Authorization callback URL field even for anonymous authentication - I found any url value would work for this. At the end you should have something similar to:

Screen shot of an Imgur completed registration form

Press on the submit button and on the next screen you should see your unique client-id value - store this somewhere safe and private as you will need this later on.

Looking at the project

Now that you have the client-id, head on over to my GitHub account where you can download the example project. You will need to add your client-id as the value of the clientID property which is found in the RequestConfig class. After you've added your client-id, play around with the app for a bit before we continue.

cat-waiting

If everything worked well, you should be looking at the newest selection of Imgur posts containing cats (or at least posts tagged as containing cats).

The example project contains classes around retrieving the Imgur feed and populating the UI but as this post is more concerned with building a media download system rather than building an Imgur app we will focus on the following classes:

  • AssetDownloadManager
  • AssetDownloadItem

AssetDownloadManager acts as a scheduler/manager of asset download requests ensuring that the most important downloads are given the most bandwidth possible and AssetDownloadItem is our asset download abstraction.

AssetDownloadItem

Ok so let's go through some code.

Each download request can be thought of as one unit of work:

enum DataRequestResult<T> {
    case success(T)
    case failure(Error)
}

typealias AssetDownloadItemCompletionHandler = ((_ result: DataRequestResult<Data>) -> Void)

enum Status: String {
    case waiting
    case downloading
    case suspended
    case canceled
}

class AssetDownloadItem {

    fileprivate let task: URLSessionDownloadTask

    var completionHandler: AssetDownloadItemCompletionHandler?

    var forceDownload = false
    var downloadPercentageComplete = 0.0
    var status = Status.waiting

    // MARK: - Init

    init(task: URLSessionDownloadTask) {
        self.task = task
    }

    // MARK: - Lifecycle

    func pause() {
        status = .waiting
        forceDownload = false
        task.suspend()
    }

    func resume() {
        status = .downloading
        task.resume()
    }

    func softCancel() {
        status = .suspended
        forceDownload = false
        task.suspend()
    }

    func hardCancel() {
        status = .canceled
        forceDownload = false
        task.cancel()
    }

    // MARK: - Meta

    var url: URL {
        return task.currentRequest!.url!
    }

    //MARK: - Coalesce

    func coalesce(_ additionalCompletionHandler: @escaping AssetDownloadItemCompletionHandler) {
        let initalCompletionHandler = completionHandler

        completionHandler = { (result) in
            if let initalCompletionClosure = initalCompletionHandler {
                initalCompletionClosure(result)
            }

            additionalCompletionHandler(result)
        }
    }
}

extension AssetDownloadItem: Equatable {
    static func ==(lhs: AssetDownloadItem, rhs: AssetDownloadItem) -> Bool {
        return lhs.url == rhs.url
    }
}

As we can see the above class is a simple data model wrapper around an URLSessionDownloadTask instance, providing that task with additional state and an interface that better suits our needs.

Lets explore this class a bit deeper.

enum DataRequestResult<T> {
    case success(T)
    case failure(Error)
}

The result enum pattern is turning into a fairly standard approach to handling any type of request that can fail, please see this post for more details on it if it's unfamiliar.

fileprivate let task: URLSessionDownloadTask

var completionHandler: AssetDownloadItemCompletionHandler?

var forceDownload = false
var downloadPercentageComplete = 0.0
var status = Status.waiting

Lets look at two of those properties:

completionHandler is a closure/block that will hold what should happen when our download task is complete - either successfully or not.

forceDownload will allow us to treat AssetDownloadItem instances differently by giving them a priority value - forced or not. A forced download, as we will see in AssetDownloadManager, effectively stops other downloads and cause that AssetDownloadItem instance to be executed (start downloading) immediately.

The lifecycle of an AssetDownloadItem instance mirrors closely the lifecycle of it's URLSessionDownloadTask instance:

func resume() {
    status = .downloading
    task.resume()
}

func pause() {
    status = .waiting
    forceDownload = false
    task.suspend()
}

func softCancel() {
    status = .suspended
    forceDownload = false
    task.suspend()
}

func hardCancel() {
    status = .canceled
    forceDownload = false
    task.cancel()
}

Let's go through these methods one at time.

func resume() {
    status = .downloading
    task.resume()
}

resume is a simple method that calls through to the resume method on the wrapped URLSessionDownloadTask instance causing the download to begin/resume. It also sets the status property allowing for accurate querying of the status of the dowonload item.

func pause() {
    status = .waiting
    forceDownload = false
    task.suspend()
}

pause is similar to the resume method we seen before. Again pause calls through to the wrapped URLSessionDownloadTask instance and also sets it's forced download value to false. This is because once an item is paused, it is an indication that the download while still valued is no longer to be treated as urgent.

An interesting point is that we have two cancel methods: softCancel and hardCancel.

func softCancel() {
    status = .suspended
    forceDownload = false
    task.suspend()
}

func hardCancel() {
    status = .canceled
    forceDownload = false
    task.cancel()
}

softCancel is similar to the pause method, in that we suspend the wrapped URLSessionDownloadTask. This allows us to revive that download-item if the user triggers a request with the same URL. We will see how this revival is possible below in the coalesce method below.

hardCancel actually cancels the wrapped URLSessionDownloadTask. This causes any downloaded content to to be thrown away.

func coalesce(_ additionalCompletionHandler: @escaping AssetDownloadItemCompletionHandler) {
    let initalCompletionHandler = completionHandler

    completionHandler = { (result) in
        if let initalCompletionClosure = initalCompletionHandler {
            initalCompletionClosure(result)
        }

        additionalCompletionHandler(result)
    }
}

As we touched on above: multiple scheduled download attempts to the same URL will not result in multiple requests actually being triggered. Instead these duplicate requests will be coalesced. This is achieved by taking the subsequent completionHandler callback and attaching it to the first download-item's completionHandler. Which actually means that any call to the first item's completionHandler will result in all callbacks being recursively called. This ability to coalesce will ensure that we make the minimum number of network requests regardless of how indecisive our users are.

AssetDownloadManager

Now we have looked in-depth at our download model class AssetDownloadItem, let's look at the class acts as a manager/scheduler of those download-items.

class AssetDownloadManager: NSObject {

    var waiting = [AssetDownloadItem]()
    var downloading = [AssetDownloadItem]()
    var suspended = [AssetDownloadItem]()

    static var maximumConcurrentDownloadsResetValue = Int.max

    var maximumConcurrentDownloads = AssetDownloadManager.maximumConcurrentDownloadsResetValue

    lazy var urlSession: URLSession = {
        let configuration = URLSessionConfiguration.background(withIdentifier: UUID().uuidString)
        let session = URLSession(configuration: configuration, delegate: self, delegateQueue: nil)

        return session
    }()

    var shouldGenerateReport = false

    // MARK: - Singleton

    static let shared = AssetDownloadManager()

    // MARK: - Init

    init(notificationCenter: NotificationCenter = NotificationCenter.default) {
        super.init()
        notificationCenter.addObserver(forName: Notification.Name.UIApplicationDidReceiveMemoryWarning, object: nil, queue: OperationQueue.main) {_ in
            for downloadAssetItem in self.suspended {
                downloadAssetItem.hardCancel()
            }

            self.suspended.removeAll()
        }
    }

    // MARK: - Schedule

    func scheduleDownload(url: URL, forceDownload: Bool, completionHandler: @escaping AssetDownloadItemCompletionHandler) {
        download(url: url, forceDownload: forceDownload, completionHandler: completionHandler)
    }

    // MARK: - Download

    private func download(url: URL, forceDownload: Bool, completionHandler: @escaping AssetDownloadItemCompletionHandler) {
        if forceDownload {
            pauseDownloads()
        }

        if let (_, assetDownloadItem) = searchForDownloadingAssetDownloadItem(withURL: url) {
            coalesceSameURLAssetDownloads(assetDownloadItem: assetDownloadItem, forceDownload: forceDownload, completionHandler: completionHandler)
        } else if let (index, assetDownloadItem) = searchForWaitingAssetDownloadItem(withURL: url) {
            coalesceSameURLAssetDownloads(assetDownloadItem: assetDownloadItem, forceDownload: forceDownload, completionHandler: completionHandler)
            waiting.remove(at: index)
            waiting.append(assetDownloadItem) // Move download-item to the front of the waiting stack
        } else if let (index, assetDownloadItem) = searchForCanceledAssetDownloadItem(withURL: url) {
            //as it's canceled, no need to coalesce
            assetDownloadItem.completionHandler = completionHandler
            assetDownloadItem.forceDownload = forceDownload

            waiting.append(assetDownloadItem)
            suspended.remove(at: index)
        } else {
            let downloadTask = urlSession.downloadTask(with: url)
            let assetDownloadItem = AssetDownloadItem(task: downloadTask)
            assetDownloadItem.completionHandler = completionHandler
            assetDownloadItem.forceDownload = forceDownload

            waiting.append(assetDownloadItem)
        }

        resumeDownloads()
    }

    private func pauseDownloads() {
        for assetDownloadItem in downloading.reversed() {
            assetDownloadItem.pause()
            waiting.append(assetDownloadItem)
        }

        downloading.removeAll()
    }

    private func resumeDownloads() {
        updateConcurrentDownloadLimitIfNeeded()

        for _ in downloading.count..<maximumConcurrentDownloads {
            guard let assetDownloadItem = waiting.last else {
                return
            }

            waiting.removeLast()
            downloading.append(assetDownloadItem)
            assetDownloadItem.resume()
        }

        generateReport()
    }

    func cancelDownload(url: URL) {
        var assetDownloadItemToBeSuspended: AssetDownloadItem?

        for (index, assetDownloadItem) in downloading.enumerated() {
            if assetDownloadItem.url == url {
                assetDownloadItemToBeSuspended = assetDownloadItem
                downloading.remove(at: index)
                break
            }
        }

        if assetDownloadItemToBeSuspended == nil {
            for (index, assetDownloadItem) in waiting.enumerated() {
                if assetDownloadItem.url == url {
                    assetDownloadItemToBeSuspended = assetDownloadItem
                    waiting.remove(at: index)
                    break
                }
            }
        }

        if let assetDownloadItem = assetDownloadItemToBeSuspended {
            assetDownloadItem.softCancel()
            suspended.append(assetDownloadItem)
        }

        generateReport()

        resumeDownloads()
    }

    // MARK: - Limit

    private func updateConcurrentDownloadLimitIfNeeded() {
        guard let waitingAssetDownloadItem = waiting.last, waitingAssetDownloadItem.forceDownload == true else {
            maximumConcurrentDownloads = AssetDownloadManager.maximumConcurrentDownloadsResetValue
            return
        }
        
        maximumConcurrentDownloads = 1
    }

    // MARK: - Coalesce

    private func coalesceSameURLAssetDownloads(assetDownloadItem: AssetDownloadItem, forceDownload: Bool, completionHandler: @escaping AssetDownloadItemCompletionHandler) {
        assetDownloadItem.coalesce(completionHandler)
        if forceDownload { //Only care about upgrading forced value
            assetDownloadItem.forceDownload = forceDownload
        }
    }

    // MARK: - Search

    private func searchForAssetDownloadItem(withURL url: URL, in array: [AssetDownloadItem]) -> (Int, AssetDownloadItem)? {
        for (index, assetDownloadItem) in array.enumerated() {
            if assetDownloadItem.url == url {
                return (index, assetDownloadItem)
            }
        }

        return nil
    }

    fileprivate func searchForWaitingAssetDownloadItem(withURL url: URL) -> (Int, AssetDownloadItem)? {
        return searchForAssetDownloadItem(withURL: url, in: waiting)
    }

    fileprivate func searchForDownloadingAssetDownloadItem(withURL url: URL) -> (Int, AssetDownloadItem)? {
        return searchForAssetDownloadItem(withURL: url, in: downloading)
    }

    fileprivate func searchForCanceledAssetDownloadItem(withURL url: URL) -> (Int, AssetDownloadItem)? {
        return searchForAssetDownloadItem(withURL: url, in: suspended)
    }

    fileprivate func searchForAssetDownloadItem(withURLSessionTask sessionTask: URLSessionTask) -> (Int, AssetDownloadItem)? {
        guard let url = sessionTask.currentRequest?.url else {
            return nil
        }

        return searchForAssetDownloadItem(withURL: url, in: downloading)
    }
}

extension AssetDownloadManager: URLSessionDownloadDelegate {

    // MARK: - URLSessionDownloadDelegate

    func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didWriteData bytesWritten: Int64, totalBytesWritten: Int64, totalBytesExpectedToWrite: Int64) {
        guard let (_, assetDownloadItem) = searchForAssetDownloadItem(withURLSessionTask: downloadTask)  else {
            return
        }

        let percentageComplete = Double(totalBytesWritten)/Double(totalBytesExpectedToWrite)
        assetDownloadItem.downloadPercentageComplete = percentageComplete

        generateReport()
    }

    func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didFinishDownloadingTo location: URL) {
        guard let (index, assetDownloadItem) = searchForAssetDownloadItem(withURLSessionTask: downloadTask) else {
            return
        }

        self.downloading.remove(at: index)
        self.resumeDownloads()

        do {
            let data = try Data(contentsOf: location)

            assetDownloadItem.completionHandler?(.success(data))
        } catch {
            assetDownloadItem.completionHandler?(.failure(APIError.invalidData))
        }

        generateReport()
    }

    func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
        guard let (index, assetDownloadItem) = searchForAssetDownloadItem(withURLSessionTask: task)  else {
            return
        }

        self.downloading.remove(at: index)
        self.resumeDownloads()

        guard let error = error else {
            assetDownloadItem.completionHandler?(.failure(APIError.unknown))
            return
        }

        assetDownloadItem.completionHandler?(.failure(error))

        generateReport()
    }
}

extension AssetDownloadManager {

    // MARK: - Report

    private func generateReport() {
        guard shouldGenerateReport else {
            return
        }

        print("-------------------------------")
        print("-       Download Report       -")
        print("-------------------------------")
        print("Date: \(Date())")
        print("Number of possible concurrent downloads: \(maximumConcurrentDownloads)")
        print("Number of items downloading: \(downloading.count)")
        print("Number of items waiting for download: \(waiting.count)")
        print("Number of items canceled for download: \(suspended.count)")

        print("")
        print("Items:")

        let combined = downloading + waiting + suspended

        if combined.count > 0 {
            for assetDownloadItem in combined {
                let percentage = (assetDownloadItem.downloadPercentageComplete*10000).rounded()/100
                print("\(assetDownloadItem.url.absoluteString) (\(percentage)%) \(assetDownloadItem.status.rawValue) \(assetDownloadItem.forceDownload ? "forced" : "unforced")")
            }
        } else {
            print("Empty")
        }

        print("")
    }
}

OK, that's a lot of code. Let's look into what it's doing.

var waiting = [AssetDownloadItem]()
var downloading = [AssetDownloadItem]()
var suspended = [AssetDownloadItem]()

As alluded to in the AssetDownloadItem class, a download-item can fall into 3 possible categories: waiting, downloading and suspended, with each array connected to those categories.

The waiting array is being used as a stack that contains those download-items that are scheduled for download but that are not actively being downloaded - due to the downloading queue having reached it's capacity. When a space opens up on the downloading queue, the item at the top of the stack will be automatically popped and added to the downloading queue. The waiting array is a stack because the system assumes that newer download requests are more important to the user than older download requests.

The downloading array is being used as a queue that contains those download-items that are currently being downloaded. The downloading as either a concurrent or serial queue, as we will see shortly this behaviour is controlled by the presence or absence of a forceDownload download-item in the system.

The suspended array is actually being used as an array that contains those download-items which have been soft-canceled (Note that we don't hold onto hard-cancelled download-items, as these items should be dealloc'd at the first opportunity by the system). Any download-item in the suspended array can be revived due the user's actions and added to either the waiting stack or downloading queue but will never be automatically moved to either waiting or downloading even if they are empty.

static var maximumConcurrentDownloadsResetValue = Int.max

var maximumConcurrentDownloads = AssetDownloadManager.maximumConcurrentDownloadsResetValue

It's important to note that the way this download system is designed there can be only one forced download present - if a forced download is active and another forced download is added, that first download will be paused, moved onto the waiting stack and have it's forceDownload property set to false. This means that maximumConcurrentDownloads can only be either Int.max or 1. I toyed with idea of setting maximumConcurrentDownloadsResetValue to be arbitrary value like 4 but in the end I think it's better to leave this up to the underlaying URLSession to determine how many downloads it can simultaneously handle with any overspill being queued by the URLSession itself until network resources become available. What this means is that just because a download-item is in the downloading queue it may not be actually downloading.

lazy var urlSession: URLSession = {
    let configuration = URLSessionConfiguration.background(withIdentifier: UUID().uuidString)
    let session = URLSession(configuration: configuration, delegate: self, delegateQueue: nil)

    return session
}()

In order to be able to update the waiting and downloading arrays we need to take greater control of the downloading process. We do this by setting the manager (AssetDownloadManager) as the delegate of the custom URLSession which our download will be scheduled on.

static let shared = AssetDownloadManager()

In the example project, I had to set AssetDownloadManager up as a singleton to ensure that all downloads were being scheduled on the same URLSession with the same waiting, downloading and suspended arrays being used.

init(notificationCenter: NotificationCenter = NotificationCenter.default) {
    super.init()
    notificationCenter.addObserver(forName: Notification.Name.UIApplicationDidReceiveMemoryWarning, object: nil, queue: OperationQueue.main) {_ in
        for downloadAssetItem in self.suspended {
            downloadAssetItem.hardCancel()
        }

        self.suspended.removeAll()
    }
}

When we cancel a download-item we are really soft-cancelling which results in that download-item's URLSessionDownloadTask being put into a suspended state. Soft-cancelling comes with the advantage in that we can resume a download from where it was paused/suspended. Each suspended download consumes some of our app's resources; eventually if we have too many suspended, iOS will trigger a UIApplicationDidReceiveMemoryWarning notification - which is a warning that unless we do something to lower our memory consumption, iOS will kill our app. In this case that something is looping through the suspended array and hard-cancelling each download-item. This will cause any downloaded data to be discarded and so lower our memory consumption. Finally, we clear the suspended array causing those suspended AssetDownloadItem instances to be dealloc'd.

func scheduleDownload(url: URL, forceDownload: Bool, completionHandler: @escaping AssetDownloadItemCompletionHandler) {
    download(url: url, forceDownload: forceDownload, completionHandler: completionHandler)
}

private func download(url: URL, forceDownload: Bool, completionHandler: @escaping AssetDownloadItemCompletionHandler) {
    if forceDownload {
        pauseDownloads()
    }

    if let (_, assetDownloadItem) = searchForDownloadingAssetDownloadItem(withURL: url) {
        coalesceSameURLAssetDownloads(assetDownloadItem: assetDownloadItem, forceDownload: forceDownload, completionHandler: completionHandler)
    } else if let (index, assetDownloadItem) = searchForWaitingAssetDownloadItem(withURL: url) {
        coalesceSameURLAssetDownloads(assetDownloadItem: assetDownloadItem, forceDownload: forceDownload, completionHandler: completionHandler)
        waiting.remove(at: index)
        waiting.append(assetDownloadItem) // Move download-item to the front of the waiting stack
    } else if let (index, assetDownloadItem) = searchForCanceledAssetDownloadItem(withURL: url) {
        //as it's canceled, no need to coalesce
        assetDownloadItem.completionHandler = completionHandler
        assetDownloadItem.forceDownload = forceDownload

        waiting.append(assetDownloadItem)
        suspended.remove(at: index)
    } else {
        let downloadTask = urlSession.downloadTask(with: url)
        let assetDownloadItem = AssetDownloadItem(task: downloadTask)
        assetDownloadItem.completionHandler = completionHandler
        assetDownloadItem.forceDownload = forceDownload

        waiting.append(assetDownloadItem)
    }

    resumeDownloads()
}

private func pauseDownloads() {
    for assetDownloadItem in downloading.reversed() {
        assetDownloadItem.pause()
        waiting.append(assetDownloadItem)
    }

    downloading.removeAll()
}

private func resumeDownloads() {
    updateConcurrentDownloadLimitIfNeeded()

    for _ in downloading.count..<maximumConcurrentDownloads {
        guard let assetDownloadItem = waiting.last else {
            return
        }

        waiting.removeLast()
        downloading.append(assetDownloadItem)
        assetDownloadItem.resume()
    }

    generateReport()
}

This is the core of the downloading system. In the download method, the first thing we do is check if the to-be scheduled download is a forced download, if it is we pause any active downloads and move those items onto the waiting stack. We then need to determine if this to-be scheduled download is in fact a new download or a re-scheduling of an existing download - we do this by searching through the waiting, downloading and suspended arrays. If it is pre-existing, we coalesce the new download request with existing download-item else a new instance of AssetDownloadItem is created for that URL. Regardless, whether it's an existing or newly created download-item we push it onto the waiting stack. Finally, we call the resumeDownloads method which loops through the waiting stack and resumes those downloads. We resume download-items until the number of active downloads match the maximum possible concurrent downloads or the waiting stack is empty.

func cancelDownload(url: URL) {
    var assetDownloadItemToBeSuspended: AssetDownloadItem?

    for (index, assetDownloadItem) in downloading.enumerated() {
        if assetDownloadItem.url == url {
            assetDownloadItemToBeSuspended = assetDownloadItem
            downloading.remove(at: index)
            break
        }
    }

    if assetDownloadItemToBeSuspended == nil {
        for (index, assetDownloadItem) in waiting.enumerated() {
            if assetDownloadItem.url == url {
                assetDownloadItemToBeSuspended = assetDownloadItem
                waiting.remove(at: index)
                break
            }
        }
    }

    if let assetDownloadItem = assetDownloadItemToBeSuspended {
        assetDownloadItem.softCancel()
        suspended.append(assetDownloadItem)
    }

    generateReport()

    resumeDownloads()
}

In the above method, we implemented how to cancel a download-item. We do this by, looping through the downloading and waiting arrays attempting to find the download-item that matches the url passed in. Once found we soft-cancel that download-item and move it into the suspended array. We also have to call the resumeDownloads method - just in case the cancelled download-item was in the downloading queue to schedule the next download to begin.

private func updateConcurrentDownloadLimitIfNeeded() {
    guard let waitingAssetDownloadItem = waiting.last, waitingAssetDownloadItem.forceDownload == true else {
        maximumConcurrentDownloads = AssetDownloadManager.maximumConcurrentDownloadsResetValue
        return
    }

    maximumConcurrentDownloads = 1
}

The above method updates the maximumConcurrentDownloads property to hold either 1 if either the downloading queue or waiting stack contains a forced-download-item, or maximumConcurrentDownloadsResetValue (in this case Int.max) if they don't.

private func coalesceSameURLAssetDownloads(assetDownloadItem: AssetDownloadItem, forceDownload: Bool, completionHandler: @escaping AssetDownloadItemCompletionHandler) {
    assetDownloadItem.coalesce(completionHandler)
    if forceDownload { //Only care about upgrading forced value
        assetDownloadItem.forceDownload = forceDownload
    }
}

Coalesces the completionHandler to an existing download-item. The interesting point in the above method is that if the new download has a better forceDownload value than the existing download-item's value, it will upgrade the existing item's value.

private func searchForAssetDownloadItem(withURL url: URL, in array: [AssetDownloadItem]) -> (Int, AssetDownloadItem)? {
     for (index, assetDownloadItem) in array.enumerated() {
         if assetDownloadItem.url == url {
             return (index, assetDownloadItem)
         }
     }

     return nil
 }

 fileprivate func searchForWaitingAssetDownloadItem(withURL url: URL) -> (Int, AssetDownloadItem)? {
     return searchForAssetDownloadItem(withURL: url, in: waiting)
 }

 fileprivate func searchForDownloadingAssetDownloadItem(withURL url: URL) -> (Int, AssetDownloadItem)? {
     return searchForAssetDownloadItem(withURL: url, in: downloading)
 }

 fileprivate func searchForCanceledAssetDownloadItem(withURL url: URL) -> (Int, AssetDownloadItem)? {
     return searchForAssetDownloadItem(withURL: url, in: suspended)
 }

 fileprivate func searchForAssetDownloadItem(withURLSessionTask sessionTask: URLSessionTask) -> (Int, AssetDownloadItem)? {
     guard let url = sessionTask.currentRequest?.url else {
         return nil
     }

     return searchForAssetDownloadItem(withURL: url, in: downloading)
 }

The above methods are a set of helpers for finding an existing download-item by comparing the existing download-item's url with the search url.

AssetDownloadManager implements the URLSessionDownloadDelegate protocol:

func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didWriteData bytesWritten: Int64, totalBytesWritten: Int64, totalBytesExpectedToWrite: Int64) {
     guard let (_, assetDownloadItem) = searchForAssetDownloadItem(withURLSessionTask: downloadTask)  else {
         return
     }

     let percentageComplete = Double(totalBytesWritten)/Double(totalBytesExpectedToWrite)
     assetDownloadItem.downloadPercentageComplete = percentageComplete

     generateReport()
 }

The above method is really only used for informational purposes - determining how much of the download has been completed.

func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didFinishDownloadingTo location: URL) {
     guard let (index, assetDownloadItem) = searchForAssetDownloadItem(withURLSessionTask: downloadTask) else {
         return
     }

     self.downloading.remove(at: index)
     self.resumeDownloads()

     do {
         let data = try Data(contentsOf: location)

         assetDownloadItem.completionHandler?(.success(data))
     } catch {
         assetDownloadItem.completionHandler?(.failure(APIError.invalidData))
     }

     generateReport()
 }
 
 func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
     guard let (index, assetDownloadItem) = searchForAssetDownloadItem(withURLSessionTask: task)  else {
         return
     }

     self.downloading.remove(at: index)
     self.resumeDownloads()

     guard let error = error else {
         assetDownloadItem.completionHandler?(.failure(APIError.unknown))
         return
     }

     assetDownloadItem.completionHandler?(.failure(error))

     generateReport()
 }

Both methods listed above effectively do the same:

  1. Remove the completed download-item form the downloading queue
  2. Schedule the next item to be downloaded
  3. Trigger the completion closure(s)/block(s)

They differ in that the successful method, returns the downloaded data where as the unsuccessful method, returns either a specific error which has been passed up from the URLSession or a generic error.

private func generateReport() {
    guard shouldGenerateReport else {
        return
    }

    print("-------------------------------")
    print("-       Download Report       -")
    print("-------------------------------")
    print("Date: \(Date())")
    print("Number of possible concurrent downloads: \(maximumConcurrentDownloads)")
    print("Number of items downloading: \(downloading.count)")
    print("Number of items waiting for download: \(waiting.count)")
    print("Number of items canceled for download: \(suspended.count)")

    print("")
    print("Items:")

    let combined = downloading + waiting + suspended

    if combined.count > 0 {
        for assetDownloadItem in combined {
            let percentage = (assetDownloadItem.downloadPercentageComplete*10000).rounded()/100
            print("\(assetDownloadItem.url.absoluteString) (\(percentage)%) \(assetDownloadItem.status.rawValue) \(assetDownloadItem.forceDownload ? "forced" : "unforced")")
        }
    } else {
        print("Empty")
    }

    print("")
}

So we have ignored any calls to generateReport in the previous methods, this is because this method is strictly only needed for this post rather than for the actual implementation of this downloading system. The generateReport is purely an informational method to allow for better insight into what the system is doing.

Hidden costs

Pausing and resuming an active download is not without a cost. Depending on which version of HTTP is being used, the HTTP connection is either going to be closed altogether (most expensive) or switched over to a different download (less expensive), either way there is a latency cost with this switch. A cost that if incurred too often will actually result in downloads taking longer. Because of this cost we need to be careful when we use forceDownload.

Only half the story 📖

This has been a fairly long post so if you've made it here, take a moment to breathe out and enjoy it 👏.

To recap we built a download stack which:

  • uses a stack to organise our download requests so that the newest requests download first
  • adjusts how many downloads are active at any one given time
  • soft cancels requests to improve performance by allowing them to be revived

Now, this download stack isn't the perfect system but it should ensure that when we determine that a user's experience is being adversely affected due to network constraints then we can take actions to rectify this i.e. forceDownload. We can can further improve performance by coupling this download stack with a few other techniques:

  • downloading the smallest possible asset required for the UI
  • predicting where the user is going and prefetching those assets
  • canceling downloads once the user moves away

In the end our users are going to have to wait for media to download, all that we can do is ensure that they are waiting due to their desire for ever increasingly media rich apps rather than our choices over how downloads are handled.

What do you think? Let me know by getting in touch on Twitter - @wibosco