Get webpage content with URLSession returns 403

When attempting to fetch webpage content via the URLSession API, an unexpected 403 error code is encountered. This comprehensive guide explores the issue and provides a solution by analyzing a unit test that successfully retrieves the webpage content. It also offers an asynchronous implementation of the solution.

The unit test effectively demonstrates the functionality of the URLSession dataTaskPublisher method, which streams the webpage content as data chunks. The test sets up an expectation, configures a URL request, and subscribes to the dataTaskPublisher. Upon completion, it asserts the expected result.

func testExample() throws {
    let exp = expectation(description: "...")
    var request = URLRequest(url: URL(string: "https://dibamovie14.top")!)
    let task = URLSession.shared.dataTaskPublisher(for: request).sink {
        print($0)
        exp.fulfill()
    } receiveValue: { data, response in
        print(response)
    }

    waitForExpectations(timeout: 4) { error in
        guard let error = error else { return }
        print(error.localizedDescription)
    }
}

To address the 403 error in the asynchronous approach, it is crucial to set the User-Agent header. However, setting the "Host" header is unnecessary, as it is mandatory only in HTTP/1.1 and should not be used in HTTP/2.

Here's an example of an asynchronous function that includes the User-Agent header:

func testExampleAsync() async throws {
    var request = URLRequest(url: URL(string: "https://dibamovie14.top")!)
    let (data, response) = try await URLSession.shared.data(for: request)
    print((response as? HTTPURLResponse)?.statusCode)
    print(String(data: data, encoding: .utf8))
}

In summary, setting the User-Agent header is a crucial step in resolving the 403 error when retrieving webpage content using the URLSession API. The asynchronous implementation provided above demonstrates how to effectively handle this scenario.