How to deal with mixed content in a website which should be secured as https?

I am building a website on server A (with domain name registered), used for people to create and run their "apps".
These "apps" are actually docker containers running on server B, in the container, there lives a small web app which can be accessed directly like:

http://IP_ADDR_OF_SERVER_B:PORT

The PORT is a random big number one which maps to the docker container.Now I can make SSL certificate working on server A, so that it works fine by accessing:

https://DOMAIN_NAME_OF_SERVER_A

The problem is, I enclosed the "apps" in iframe by accessing "http" like above, therefore my browser(Chrome) refuse to open it and report error as:

Mixed Content: The page at 'https://DOMAIN_NAME_OF_SERVER_A/xxx' was loaded over HTTPS, but requested an insecure resource 'http://IP_ADDR_OF_SERVER_B:PORT/xxx'. This request has been blocked; the content must be served over HTTPS.

So, how should I deal with such issue?
I am a full stack green hand, I'd appreciate a lot if you can share some knowledge on how to build a healthy https website while solving such problem in a proper way.

Supplementary explanation

Ok I think I just threw out the outline of the question, here goes more details.

I see it is intact and straight forward to make the iframe requests to be served with https, then it won't confuse me anymore.

However the trouble is, since all the "apps" are dynamically created/removed, it seems I'll need to prepare many certificates for each one of them.

Will self signed certificate work without being blocked or complained by the browser? Or do I have a way to serve all the "apps" with one SSL certificate?

Software environment

Server A: Running node.js website listening to port 5000 and served with Nginx proxy_pass.

server {    listen 80;    server_name DOMAIN_NAME_OF_SERVER_A;    location / {        proxy_set_header   X-Real-IP $remote_addr;        proxy_set_header   Host      $http_host;        proxy_pass     http://127.0.0.1:5000;    }}server {    listen 443;    server_name DOMAIN_NAME_OF_SERVER_A;    ssl on;    ssl_certificate /etc/nginx/ssl/DOMAIN_NAME_OF_SERVER_A.cer;    ssl_certificate_key /etc/nginx/ssl/DOMAIN_NAME_OF_SERVER_A.key;    ssl_session_timeout 5m;    location / {        proxy_set_header   X-Real-IP $remote_addr;        proxy_set_header   Host      $http_host;        proxy_pass     http://127.0.0.1:5000;    }}

Server B: Running node.js apps listening to different random big port numbers such as 50055, assigned dynamically when "apps" are created. (In fact these apps are running in docker containers while I think it doesn't matter) Can run Nginx if needed.

Server A and Server B talk with each other in public traffic.

Solution

Just as all the answers, especially the one from @eawenden, I need a reverse proxy to achieve my goal.

In addition, I did a few more things:
1. Assign a domain name to Server B for using a letsencrypt cert.
2. Proxy predefined url to specific port.

Therefore I setup a reverse proxy server using nginx on Server B, proxy all the requests like:

https://DOMAIN_NAME_OF_SERVER_B/PORT/xxx

to

https://127.0.0.1:PORT/xxx

Ps: nginx reverse proxy config on Server B

server {    listen 443;    server_name DOMAIN_NAME_OF_SERVER_B;    ssl on;    ssl_certificate     /etc/nginx/ssl/DOMAIN_NAME_OF_SERVER_B.cer;    ssl_certificate_key /etc/nginx/ssl/DOMAIN_NAME_OF_SERVER_B.key;    ssl_session_timeout 5m;    rewrite_log off;    error_log   /var/log/nginx/rewrite.error.log info;    location ~ ^/(?<port>\d+)/ {        rewrite ^/\d+?(/.*) $1 break;        proxy_pass http://127.0.0.1:$port;        proxy_http_version 1.1;        proxy_set_header Upgrade $http_upgrade;        proxy_set_header Connection "upgrade";        proxy_read_timeout 86400;    }}

Thus everything seems to be working as expected!
Thanks again to all the answerers.