The source code of this example can be downloaded from here.
Introduction
The example bellow demonstrates how to stream live video from the camera to a standalone desktop application. It implements a simple client application running in Windows Phone 8.1 Silverlight which provides a UI to preview and record the video from the camera. When a user clicks the record button the application opens connection with the service application and starts streaming. The service application receives the stream and stores it into the MP4 (MPEG-4) file.
The implementation of this scenario addresses following topics:
The implementation of this scenario addresses following topics:
- Capturing video in Windows Phone 8.1
- Streaming video across the network.
- Receiving video in a desktop application and storing it to the file.
To Run Example
- Download Eneter Messaging Framework for .NET platforms.
- Download the example project.
- Open the solution in Visual Studio and update references to:
Eneter.Messaging.Framework.dll - for .NET 4.5
Eneter.Messaging.Framework.WindowsPhone.dll - for Windows Phone 8.1 Silverlight - Figure out the IP address of your computer within your network and update the IP address in the client as well as the service source code.
- Run the service application.
- Deploy the client application to the Windows Phone device and runt it.
Capturing Video in Windows Phone 8.1
To use the camera the application needs to have permissions for following capabilities:
They can be enabled in WMAppManifest.xml (located in the Properties folder).
To capture the video Windows Phone 8.1 offers the MediaCapture class which provides functionality to preview and record the video (including audio). After instantiating the MediaCapture object needs to be initialized with proper settings (see ActivateCameraAsync()) and then to enable the preview it needs to be associated with VideoBrush (see StartPreviewAsync()).
It is also very important to ensure MediaCapture is properly disposed when not needed or when the application is suspended. Failing to do so will cause problems to other applications accessing the camera. (E.g. when I did not dispose MediaCapture I could not start the application multiple times. Following starts always failed during the camera initialization and the phone had to be rebooted.)
- ID_CAP_ISV_CAMERA - provides access to the camera.
- ID_CAP_MICROPHONE - provides access to the phone's microphone.
They can be enabled in WMAppManifest.xml (located in the Properties folder).
To capture the video Windows Phone 8.1 offers the MediaCapture class which provides functionality to preview and record the video (including audio). After instantiating the MediaCapture object needs to be initialized with proper settings (see ActivateCameraAsync()) and then to enable the preview it needs to be associated with VideoBrush (see StartPreviewAsync()).
It is also very important to ensure MediaCapture is properly disposed when not needed or when the application is suspended. Failing to do so will cause problems to other applications accessing the camera. (E.g. when I did not dispose MediaCapture I could not start the application multiple times. Following starts always failed during the camera initialization and the phone had to be rebooted.)
Streaming Video Across Network
To record the video the method StartRecordToStreamAsync(..) is called. The method takes two parameters:
- MediaEncodingProfile - specifies the video format (e.g. MP4 or WMV)
- IRandomAccessStream - specifies the stream where to write captured video.
In order to provide streaming across the network custom MessageSendingStream (derived from IRandomAccessStream) is implemented and then used as an input parameter for StartRecordToStreamAsync(..).
It does not implement the whole interface but only methods which are necessary for MediaCapture to write MPEG-4 format.
When a user starts recording MessageSendingStream is instantiated and the connection with the service is open. Then StartRecordToStreamAsync(...) with MessageSendingStream is called and captured data is sent to the service.
Since writing MP4 is not fully sequential the streaming message sent from the client to the service consists of two parts:
When the user stops recording MediaCapture completes writing and the connection with the service is closed. Once the connection is closed the service closes the MP4 file.
It does not implement the whole interface but only methods which are necessary for MediaCapture to write MPEG-4 format.
When a user starts recording MessageSendingStream is instantiated and the connection with the service is open. Then StartRecordToStreamAsync(...) with MessageSendingStream is called and captured data is sent to the service.
Since writing MP4 is not fully sequential the streaming message sent from the client to the service consists of two parts:
- 4 bytes - integer number indicating the position inside MP4.
- n bytes - video/audio data captured by the camera.
When the user stops recording MediaCapture completes writing and the connection with the service is closed. Once the connection is closed the service closes the MP4 file.
Receiving Video in Desktop Application
Receiving the stream is quite straight forward. When the service receives the stream message it decodes the position (first 4 bytes) and writes incoming video data to the MP4 file on desired position.
There can be multiple recording clients connected to the service. Therefore the service maintains for each connected client a separate MP4 file. Once the client disconnects the file is closed and ready for further using (e.g. cutting or replaying).
Windows Phone Client
Windows phone client is a simple application displaying the video preview and providing buttons to start and stop the video capturing.
When the user clicks start record it opens connection with the service and starts sending stream messages.
When the user clicks stop recording or the application is suspended it completes the recording and closes the connection with the service.
The implementation consists of two major parts:
- Logic manipulating the camera - implemented in MainPage.xaml.cs file
- Logic sending the stream messages to the service - implemented in MessageSendingStream.cs file.
using Microsoft.Phone.Controls;
using Microsoft.Phone.Shell;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using System.Windows.Media;
using System.Windows.Navigation;
using Windows.Devices.Enumeration;
using Windows.Media.Capture;
using Windows.Media.MediaProperties;
using Windows.Phone.Media.Capture;
namespace PhoneCamera
{
public partial class MainPage : PhoneApplicationPage
{
// Stream sending messages to the service.
private MessageSendingStream myMessageSendingStream;
// Video capturing.
private VideoBrush myVideoRecorderBrush;
private MediaCapturePreviewSink myPreviewSink;
private MediaCapture myMediaCapture;
private MediaEncodingProfile myProfile;
private bool myIsRecording;
// Constructor
public MainPage()
{
InitializeComponent();
// Prepare ApplicationBar and buttons.
PhoneAppBar = (ApplicationBar)ApplicationBar;
PhoneAppBar.IsVisible = true;
StartRecordingBtn = ((ApplicationBarIconButton)ApplicationBar.Buttons[0]);
StopRecordingBtn = ((ApplicationBarIconButton)ApplicationBar.Buttons[1]);
}
protected override async void OnNavigatedTo(NavigationEventArgs e)
{
base.OnNavigatedTo(e);
// Disable both buttons until initialization is completed.
StartRecordingBtn.IsEnabled = false;
StopRecordingBtn.IsEnabled = false;
try
{
await ActivateCameraAsync();
await StartPreviewAsync();
// Enable Start Recording button.
StartRecordingBtn.IsEnabled = true;
txtDebug.Text = "Ready...";
}
catch (Exception err)
{
txtDebug.Text = "ERROR: " + err.Message;
}
}
protected override void OnNavigatedFrom(NavigationEventArgs e)
{
DeactivateCamera();
base.OnNavigatedFrom(e);
}
private async void OnStartRecordingClick(object sender, EventArgs e)
{
await StartRecordingAsync();
}
// Handle stop requests.
private async void OnStopRecordingClick(object sender, EventArgs e)
{
await StopRecordingAsync("Ready...");
}
private async Task ActivateCameraAsync()
{
// Find the camera device id to use
string aDeviceId = "";
var aDevices = await DeviceInformation.FindAllAsync(DeviceClass.VideoCapture);
for (var i = 0; i < aDevices.Count; ++i)
{
aDeviceId = aDevices[i].Id;
}
// Capture settings.
var aSettings = new MediaCaptureInitializationSettings();
aSettings.AudioDeviceId = "";
aSettings.VideoDeviceId = aDeviceId;
aSettings.MediaCategory = MediaCategory.Other;
aSettings.PhotoCaptureSource = PhotoCaptureSource.VideoPreview;
aSettings.StreamingCaptureMode = StreamingCaptureMode.AudioAndVideo;
//Create profile for MPEG-4 container which will have H.264 video.
myProfile = MediaEncodingProfile.CreateMp4(
Windows.Media.MediaProperties.VideoEncodingQuality.Qvga);
// Initialize the media capture with specified settings.
myMediaCapture = new MediaCapture();
await myMediaCapture.InitializeAsync(aSettings);
myIsRecording = false;
}
private void DeactivateCamera()
{
if (myMediaCapture != null)
{
if (myIsRecording)
{
// Note: Camera deactivation needs to run synchronous.
// Otherwise suspending/resuming during recording does not work.
myMediaCapture.StopRecordAsync().AsTask().Wait();
myIsRecording = false;
}
myMediaCapture.StopPreviewAsync().AsTask().Wait();
myMediaCapture.Dispose();
myMediaCapture = null;
}
if (myPreviewSink != null)
{
myPreviewSink.Dispose();
myPreviewSink = null;
}
ViewfinderRectangle.Fill = null;
if (myMessageSendingStream != null)
{
myMessageSendingStream.CloseConnection();
myMessageSendingStream.ConnectionBroken -= OnConnectionBroken;
}
}
private async void OnConnectionBroken(object sender, EventArgs e)
{
await StopRecordingAsync("Disconnected from server.");
}
private async Task StartPreviewAsync()
{
// List of supported video preview formats to be used by the default
// preview format selector.
var aSupportedVideoFormats = new List<string> { "nv12", "rgb32" };
// Find the supported preview format
var anAvailableMediaStreamProperties =
myMediaCapture.VideoDeviceController.GetAvailableMediaStreamProperties(
Windows.Media.Capture.MediaStreamType.VideoPreview)
.OfType<Windows.Media.MediaProperties.VideoEncodingProperties>()
.Where(p => p != null && !String.IsNullOrEmpty(p.Subtype)
&& aSupportedVideoFormats.Contains(p.Subtype.ToLower()))
.ToList();
var aPreviewFormat = anAvailableMediaStreamProperties.FirstOrDefault();
// Start Preview stream
myPreviewSink = new MediaCapturePreviewSink();
await myMediaCapture.VideoDeviceController.SetMediaStreamPropertiesAsync(
Windows.Media.Capture.MediaStreamType.VideoPreview, aPreviewFormat);
await myMediaCapture.StartPreviewToCustomSinkAsync(
new MediaEncodingProfile { Video = aPreviewFormat }, myPreviewSink);
// Create the VideoBrush for the viewfinder.
myVideoRecorderBrush = new VideoBrush();
Microsoft.Devices.CameraVideoBrushExtensions.SetSource(myVideoRecorderBrush, myPreviewSink);
// Display video preview.
ViewfinderRectangle.Fill = myVideoRecorderBrush;
}
private async Task StartRecordingAsync()
{
// Disable Start Recoridng button.
StartRecordingBtn.IsEnabled = false;
try
{
// Connect the service.
// Note: use IP address of the service within your network.
// To figure out the IP address of you can execute from
// the command prompt: ipconfig -all
myMessageSendingStream =
new MessageSendingStream("tcp://192.168.178.31:8093/");
myMessageSendingStream.ConnectionBroken += OnConnectionBroken;
myMessageSendingStream.OpenConnection();
await myMediaCapture.StartRecordToStreamAsync(myProfile, myMessageSendingStream);
myIsRecording = true;
// Enable Stop Recording button.
StopRecordingBtn.IsEnabled = true;
txtDebug.Text = "Recording...";
}
catch (Exception err)
{
myMessageSendingStream.CloseConnection();
myMessageSendingStream.ConnectionBroken -= OnConnectionBroken;
txtDebug.Text = "ERROR: " + err.Message;
StartRecordingBtn.IsEnabled = true;
}
}
private async Task StopRecordingAsync(string textMessage)
{
// Note: Since this method does not have to be called from UI thread
// ensure UI controls are manipulated from the UI thread.
// Disable Stop Recording button.
ToUiThread(() => StopRecordingBtn.IsEnabled = false);
try
{
if (myIsRecording)
{
await myMediaCapture.StopRecordAsync();
myIsRecording = false;
}
// Enable Start Recording button and display Message.
ToUiThread(() =>
{
StartRecordingBtn.IsEnabled = true;
txtDebug.Text = textMessage;
});
}
catch (Exception err)
{
ToUiThread(() =>
{
txtDebug.Text = "ERROR: " + err.Message;
StopRecordingBtn.IsEnabled = true;
});
}
// Disconnect from the service.
myMessageSendingStream.CloseConnection();
myMessageSendingStream.ConnectionBroken -= OnConnectionBroken;
}
private void ToUiThread(Action x)
{
Dispatcher.BeginInvoke(x);
}
}
}
Implementation of MessageSendingStream.cs is very simple too:
using Eneter.Messaging.MessagingSystems.ConnectionProtocols;
using Eneter.Messaging.MessagingSystems.MessagingSystemBase;
using Eneter.Messaging.MessagingSystems.TcpMessagingSystem;
using System;
using System.Runtime.InteropServices.WindowsRuntime;
using System.Threading;
using System.Threading.Tasks;
using Windows.Foundation;
using Windows.Storage.Streams;
namespace PhoneCamera
{
// Implements IRandomAccessStream which is then used by MediaCapture for storing
// captured video and audi data.
// The implementation of this class sends the captured data across the network
// to the service where it is stored to the file.
internal class MessageSendingStream : IRandomAccessStream
{
private IDuplexOutputChannel myOutputChannel;
private ulong myPosition; // Current position in the stream.
private ulong mySize; // The size of the stream.
// Raised when the connection with the service is broken.
public event EventHandler ConnectionBroken;
public MessageSendingStream(string serviceAddress)
{
// Let's use TCP for the communication with fast encoding of messages.
var aFormatter = new EasyProtocolFormatter();
var aMessaging = new TcpMessagingSystemFactory(aFormatter);
aMessaging.ConnectTimeout = TimeSpan.FromMilliseconds(3000);
myOutputChannel = aMessaging.CreateDuplexOutputChannel(serviceAddress);
}
public void OpenConnection()
{
myOutputChannel.OpenConnection();
}
public void CloseConnection()
{
myOutputChannel.CloseConnection();
}
public bool CanRead { get { return false; } }
public bool CanWrite { get { return true; } }
public IRandomAccessStream CloneStream()
{ throw new NotSupportedException(); }
public IInputStream GetInputStreamAt(ulong position)
{ throw new NotSupportedException(); }
public IOutputStream GetOutputStreamAt(ulong position)
{ throw new NotSupportedException(); }
public ulong Position { get { return myPosition; } }
public void Seek(ulong position)
{
myPosition = position;
if (myPosition >= mySize)
{
mySize = myPosition + 1;
}
}
public ulong Size
{
get { return mySize; }
set { throw new NotSupportedException(); }
}
public void Dispose()
{
myOutputChannel.CloseConnection();
}
public IAsyncOperationWithProgress<IBuffer, uint> ReadAsync(IBuffer buffer, uint count, InputStreamOptions options)
{
throw new NotSupportedException();
}
public IAsyncOperation<bool> FlushAsync()
{
throw new NotSupportedException();
}
// Implements sending of video/audio data to the service.
// The message is encoded the following way:
// 4 bytes - position in MP4 file where data shall be put.
// n bytes - video/audio data.
public IAsyncOperationWithProgress<uint, uint> WriteAsync(IBuffer buffer)
{
Task<uint> aTask = new Task<uint>(() =>
{
uint aVideoDataLength = buffer.Length;
byte[] aMessage = new byte[aVideoDataLength + 4];
// Put position within MP4 file to the message.
byte[] aPosition = BitConverter.GetBytes((int)myPosition);
Array.Copy(aPosition, aMessage, aPosition.Length);
// Put video/audio data to the message.
buffer.CopyTo(0, aMessage, 4, (int)aVideoDataLength);
uint aTransferedSize = 0;
try
{
// Send the message to the service.
myOutputChannel.SendMessage(aMessage);
aTransferedSize = (uint)aVideoDataLength;
// Calculate new size of the stream.
if (myPosition + aVideoDataLength > mySize)
{
mySize = myPosition + aVideoDataLength;
}
}
catch
{
// If sending fails then the connection is broken.
if (ConnectionBroken != null)
{
ConnectionBroken(this, new EventArgs());
}
}
return aTransferedSize;
});
aTask.RunSynchronously();
Func<CancellationToken, IProgress<uint>, Task<uint>> aTaskProvider = (token, progress) => aTask;
return AsyncInfo.Run<uint, uint>(aTaskProvider);
}
}
}
Desktop Service
Desktop service is a simple console application which listens to a specified IP address and port.
When a client connects the service it creates the MP4 file and waits for stream messages. When a stream message is received it writes data to the file.
The whole implementation is very simple:
using Eneter.Messaging.MessagingSystems.ConnectionProtocols;
using Eneter.Messaging.MessagingSystems.MessagingSystemBase;
using Eneter.Messaging.MessagingSystems.TcpMessagingSystem;
using System;
using System.Collections.Generic;
using System.IO;
namespace VideoStorageService
{
class Program
{
// The service can handle multiple clients.
// So this dictionary maintains open files per client.
private static Dictionary<string, FileStream> myActiveVideos = new Dictionary<string, FileStream>();
static void Main(string[] args)
{
var aFastEncoding = new EasyProtocolFormatter();
var aMessaging = new TcpMessagingSystemFactory(aFastEncoding);
var anInputChannel = aMessaging.CreateDuplexInputChannel("tcp://192.168.178.31:8093/");
// Subscribe to handle incoming data.
anInputChannel.MessageReceived += OnMessageReceived;
// Subscribe to handle client connection/disconnection.
anInputChannel.ResponseReceiverConnected += OnResponseReceiverConnected;
anInputChannel.ResponseReceiverDisconnected += OnClientDisconnected;
// Start listening.
anInputChannel.StartListening();
Console.WriteLine("Videostorage service is running. Press ENTER to stop.");
Console.WriteLine("The service is listening at: " + anInputChannel.ChannelId);
Console.ReadLine();
// Stop listening.
// Note: it releases the listening thread.
anInputChannel.StopListening();
}
private static void OnResponseReceiverConnected(object sender, ResponseReceiverEventArgs e)
{
Console.WriteLine("Connected client: " + e.ResponseReceiverId);
StartStoring(e.ResponseReceiverId);
}
private static void OnClientDisconnected(object sender, ResponseReceiverEventArgs e)
{
Console.WriteLine("Disconnected client: " + e.ResponseReceiverId);
StopStoring(e.ResponseReceiverId);
}
private static void OnMessageReceived(object sender, DuplexChannelMessageEventArgs e)
{
byte[] aVideoData = (byte[])e.Message;
StoreVideoData(e.ResponseReceiverId, aVideoData);
}
private static void StartStoring(string clientId)
{
// Create MP4 file for the client.
string aFileName = "./" + Guid.NewGuid().ToString() + ".mp4";
myActiveVideos[clientId] = new FileStream(aFileName, FileMode.CreateNew, FileAccess.ReadWrite, FileShare.None);
}
private static void StopStoring(string clientId)
{
// Close MP4 file for the client.
FileStream aFileStream;
myActiveVideos.TryGetValue(clientId, out aFileStream);
if (aFileStream != null)
{
aFileStream.Close();
}
myActiveVideos.Remove(clientId);
}
private static void StoreVideoData(string clientId, byte[] videoData)
{
try
{
// Get MP4 file which is open for the client.
FileStream aFileStream;
myActiveVideos.TryGetValue(clientId, out aFileStream);
if (aFileStream != null)
{
// From first 4 bytes decode position in MP4 file
// where to write the video data.
int aPosition = BitConverter.ToInt32(videoData, 0);
// Set the position in the file.
aFileStream.Seek(aPosition, SeekOrigin.Begin);
// Write data to the file.
aFileStream.Write(videoData, 4, videoData.Length - 4);
}
}
catch (Exception err)
{
Console.WriteLine(err);
}
}
}
}
No comments:
Post a Comment