Debugging a nuget package built by Azure Devops doesn't work

Let me start by saying if I build the package locally and publish it to a local drive, everything works as expected. If the same steps are done in Azure DevOps and the package is published to our priv