视频转场

视频转场，顾名思义就是由一个视频过渡到另外一个视频，通过添加一定的图像处理效果，让两个视频之间的转场更加顺畅、切合用户需要。

首先先回顾以下视频合成的流程

  1. 获取视频资源AVAsset。
  2. 创建自定义合成对象AVMutableComposition。
  3. 创建视频组件AVMutableVideoComposition，这个类是处理视频中要编辑的东西。可以设定所需视频的大小、规模以及帧的持续时间。以及管理并设置视频组件的指令。
  4. 创建遵循AVVideoCompositing协议的customVideoCompositorClass，这个类主要用于定义视频合成器的属性和方法。
  5. 在可变组件中添加资源数据，也就是轨道AVMutableCompositionTrack（一般添加2种：音频轨道和视频轨道）。
  6. 创建视频应用层的指令AVMutableVideoCompositionLayerInstruction 用户管理视频框架应该如何被应用和组合，也就是说是子视频在总视频中出现和消失的时间、大小、动画等。
  7. 创建视频导出会话对象AVAssetExportSession，主要是根据 videoComposition 去创建一个新的视频，并输出到一个指定的文件路径中去。

构建转场AVMutableVideoCompositionLayerInstruction

视频转场，首要条件是获取到需要添加转场的两个视频的视频帧数据。所以我们在构建AVMutableVideoCompositionLayerInstruction的时候，添加一段转场所需的AVMutableVideoCompositionLayerInstruction

  1. - (void)buildTransitionComposition:(AVMutableComposition *)composition andVideoComposition:(AVMutableVideoComposition *)videoComposition {
  2.     NSUInteger clipsCount = self.clips.count;
  3.     CMTime nextClipStartTime = kCMTimeZero;
  5.     /// 转场时间为2s
  6.     CMTime transitionDuration = CMTimeMakeWithSeconds(2, 600);
  8.     // Add two video tracks and two audio tracks.
  9.     AVMutableCompositionTrack *compositionVideoTracks[2];
  10.     AVMutableCompositionTrack *compositionAudioTracks[2];
  11.     compositionVideoTracks[0] = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];
  12.     compositionVideoTracks[1] = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];
  13.     compositionAudioTracks[0] = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid];
  14.     compositionAudioTracks[1] = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid];
  16.     CMTimeRange *timeRanges = alloca(sizeof(CMTimeRange) * clipsCount);
  17.     CMTimeRange *transitionTimeRanges = alloca(sizeof(CMTimeRange) * clipsCount);
  19.     // Place clips into alternating video & audio tracks in composition
  20.     for (int i = 0; i < clipsCount; i++) {
  21.         AVAsset *asset = [self.clips objectAtIndex:i];
  22.         CMTimeRange timeRange = CMTimeRangeMake(kCMTimeZero, [asset duration]);
  24.         AVAssetTrack *clipVideoTrack = [[asset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
  25.         [compositionVideoTracks[i] insertTimeRange:timeRange ofTrack:clipVideoTrack atTime:nextClipStartTime error:nil];
  27.         AVAssetTrack *clipAudioTrack = [[asset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0];
  28.         [compositionAudioTracks[i] insertTimeRange:timeRange ofTrack:clipAudioTrack atTime:nextClipStartTime error:nil];
  30.         timeRanges[i] = CMTimeRangeMake(nextClipStartTime, timeRange.duration);
  31.         /// 根据转场时间，相对应的剪短当前视频时长
  32.         if (i > 0) {
  33.             timeRanges[i].start = CMTimeAdd(timeRanges[i].start, transitionDuration);
  34.             timeRanges[i].duration = CMTimeSubtract(timeRanges[i].duration, transitionDuration);
  35.         }
  36.         if (i+1 < clipsCount) {
  37.             timeRanges[i].duration = CMTimeSubtract(timeRanges[i].duration, transitionDuration);
  38.         }
  40.         /// 更新下个 asset 开始时间
  41.         nextClipStartTime = CMTimeAdd(nextClipStartTime, asset.duration);
  42.         nextClipStartTime = CMTimeSubtract(nextClipStartTime, transitionDuration);
  44.         /// 处理转场时间
  45.         if (i+1 < clipsCount) {
  46.             transitionTimeRanges[i] = CMTimeRangeMake(nextClipStartTime, transitionDuration);
  47.         }
  48.     }
  50.     NSMutableArray *instructions = [NSMutableArray array];
  51.     for (int i = 0; i < clipsCount; i++) {
  52.         CustomVideoCompositionInstruction *videoInstruction = [[CustomVideoCompositionInstruction alloc] initTransitionWithSourceTrackIDs:@[@(compositionVideoTracks[i].trackID)] forTimeRange:timeRanges[i]];
  53.         videoInstruction.trackID = compositionVideoTracks[i].trackID;
  54.         [instructions addObject:videoInstruction];
  56.         /// 转场
  57.         if (i+1 < clipsCount) {
  58.             CustomVideoCompositionInstruction *videoInstruction = [[CustomVideoCompositionInstruction alloc] initTransitionWithSourceTrackIDs:@[[NSNumber numberWithInt:compositionVideoTracks[0].trackID], [NSNumber numberWithInt:compositionVideoTracks[1].trackID]] forTimeRange:transitionTimeRanges[i]];
  59.             // First track -> Foreground track while compositing
  60.             videoInstruction.foregroundTrackID = compositionVideoTracks[0].trackID;
  61.             // Second track -> Background track while compositing
  62.             videoInstruction.backgroundTrackID = compositionVideoTracks[1].trackID;
  63.             [instructions addObject:videoInstruction];
  64.         }
  65.     }
  66.     videoComposition.instructions = instructions;
  67. }

我们添加了视频转场对应的AVMutableVideoCompositionLayerInstruction之后，在对应AVMutableVideoCompositionLayerInstruction的时间范围内，startVideoCompositionRequest:方法会将对应的AVMutableVideoCompositionLayerInstruction回调出来；我们可以根据对应AVMutableVideoCompositionLayerInstruction持有的trackID获取对应转场视频的视频帧，根据对应的视频帧作自定义的图像处理，进而生成转场动画。

对于图像处理可以根据自己所需选择使用 OpenGL、Metal 处理。

  1. - (void)startVideoCompositionRequest:(nonnull AVAsynchronousVideoCompositionRequest *)request {
  2.     @autoreleasepool {
  3.         dispatch_async(_renderingQueue, ^{
  4.             if (self.shouldCancelAllRequests) {
  5.                 [request finishCancelledRequest];
  6.             } else {
  7.                 NSError *err = nil;
  8.                 // Get the next rendererd pixel buffer
  9.                 CVPixelBufferRef resultPixels = [self newRenderedPixelBufferForRequest:request error:&err];
  11.                 if (resultPixels) {
  12.                     CFRetain(resultPixels);
  13.                     // The resulting pixelbuffer from OpenGL renderer is passed along to the request
  14.                     [request finishWithComposedVideoFrame:resultPixels];
  15.                     CFRelease(resultPixels);
  16.                 } else {
  17.                     [request finishWithError:err];
  18.                 }
  19.             }
  20.         });
  21.     }
  22. }
  24. - (CVPixelBufferRef)newRenderedPixelBufferForRequest:(AVAsynchronousVideoCompositionRequest *)request error:(NSError **)errOut {
  25.     CVPixelBufferRef dstPixels = nil;
  26.     CustomVideoCompositionInstruction *currentInstruction = request.videoCompositionInstruction;
  28.     float tweenFactor = factorForTimeInRange(request.compositionTime, request.videoCompositionInstruction.timeRange);
  30.     // 获取指定 track 的 pixelBuffer
  31.     if (currentInstruction.trackID) {
  32.         CVPixelBufferRef currentPixelBuffer = [request sourceFrameByTrackID:currentInstruction.trackID];
  33.         dstPixels = currentPixelBuffer;
  34.     } else {
  35.         CVPixelBufferRef currentPixelBuffer1 = [request sourceFrameByTrackID:currentInstruction.foregroundTrackID];
  36.         CVPixelBufferRef currentPixelBuffer2 = [request sourceFrameByTrackID:currentInstruction.backgroundTrackID];
  37.         dstPixels = [self.renderContext newPixelBuffer];
  39.         // ogl
  40.         //[self.oglRenderer renderPixelBuffer:dstPixels foregroundPixelBuffer:currentPixelBuffer1 backgroundPixelBuffer:currentPixelBuffer2 tweenFactor:tweenFactor];
  42.         // metal
  43.         [self.metalRenderer renderPixelBuffer:dstPixels foregroundPixelBuffer:currentPixelBuffer1 backgroundPixelBuffer:currentPixelBuffer2 tweenFactor:tweenFactor];
  44.     }
  46.     return dstPixels;
  47. }

下面简单介绍一个透明度转场案例，可以让大家清晰的了解大概的操作流程

OpenGL 转场

shader
vertex shader

  1. attribute vec4 position;
  2. attribute vec2 texCoord;
  4. varying vec2 texCoordVarying;
  6. void main()
  7. {
  8.     gl_Position = position;
  9.     texCoordVarying = texCoord;
  10. }

fragment shader

  1. uniform sampler2D Sampler;
  3. precision mediump float;
  5. varying highp vec2 texCoordVarying;
  7. void main()
  8. {
  9.     vec3 color = texture2D(Sampler, texCoordVarying).rgb;
  10.     gl_FragColor = vec4(color, 1.0);
  11. }

设置 shader 着色器

  1. - (BOOL)loadShaders {
  2.     GLuint vertShader, fragShader;
  3.     NSString *vertShaderSource, *fragShaderSource;
  4.     NSString *vertShaderPath = [[NSBundle mainBundle] pathForResource:@"transition.vs" ofType:nil];
  5.     NSString *fragShaderPath = [[NSBundle mainBundle] pathForResource:@"transition.fs" ofType:nil];
  7.     // Create the shader program.
  8.     self.program = glCreateProgram();
  10.     // Create and compile the vertex shader.
  11.     vertShaderSource = [NSString stringWithContentsOfFile:vertShaderPath encoding:NSUTF8StringEncoding error:nil];
  12.     if (![self compileShader:&vertShader type:GL_VERTEX_SHADER source:vertShaderSource]) {
  13.         NSLog(@"Failed to compile vertex shader");
  14.         return NO;
  15.     }
  17.     // Create and compile Y fragment shader.
  18.     fragShaderSource = [NSString stringWithContentsOfFile:fragShaderPath encoding:NSUTF8StringEncoding error:nil];
  19.     if (![self compileShader:&fragShader type:GL_FRAGMENT_SHADER source:fragShaderSource]) {
  20.         NSLog(@"Failed to compile fragment shader");
  21.         return NO;
  22.     }
  24.     // Attach vertex shader to programY.
  25.     glAttachShader(self.program, vertShader);
  27.     // Attach fragment shader to programY.
  28.     glAttachShader(self.program, fragShader);
  31.     // Bind attribute locations. This needs to be done prior to linking.
  32.     glBindAttribLocation(self.program, ATTRIB_VERTEX, "position");
  33.     glBindAttribLocation(self.program, ATTRIB_TEXCOORD, "texCoord");
  35.     // Link the program.
  36.     if (![self linkProgram:self.program]) {
  37.         NSLog(@"Failed to link program");
  38.         if (vertShader) {
  39.             glDeleteShader(vertShader);
  40.             vertShader = 0;
  41.         }
  42.         if (fragShader) {
  43.             glDeleteShader(fragShader);
  44.             fragShader = 0;
  45.         }
  46.         if (_program) {
  47.             glDeleteProgram(_program);
  48.             _program = 0;
  49.         }
  50.         return NO;
  51.     }
  53.     // Get uniform locations.
  54.     uniforms[UNIFORM] = glGetUniformLocation(_program, "Sampler");
  56.     // Release vertex and fragment shaders.
  57.     if (vertShader) {
  58.         glDetachShader(_program, vertShader);
  59.         glDeleteShader(vertShader);
  60.     }
  61.     if (fragShader) {
  62.         glDetachShader(_program, fragShader);
  63.         glDeleteShader(fragShader);
  64.     }
  66.     return YES;
  67. }
  69. - (BOOL)compileShader:(GLuint *)shader type:(GLenum)type source:(NSString *)sourceString
  70. {
  71.     if (sourceString == nil) {
  72.         NSLog(@"Failed to load vertex shader: Empty source string");
  73.         return NO;
  74.     }
  76.     GLint status;
  77.     const GLchar *source;
  78.     source = (GLchar *)[sourceString UTF8String];
  80.     *shader = glCreateShader(type);
  81.     glShaderSource(*shader, 1, &source, NULL);
  82.     glCompileShader(*shader);
  84. #if defined(DEBUG)
  85.     GLint logLength;
  86.     glGetShaderiv(*shader, GL_INFO_LOG_LENGTH, &logLength);
  87.     if (logLength > 0) {
  88.         GLchar *log = (GLchar *)malloc(logLength);
  89.         glGetShaderInfoLog(*shader, logLength, &logLength, log);
  90.         NSLog(@"Shader compile log:\n%s", log);
  91.         free(log);
  92.     }
  93. #endif
  95.     glGetShaderiv(*shader, GL_COMPILE_STATUS, &status);
  96.     if (status == 0) {
  97.         glDeleteShader(*shader);
  98.         return NO;
  99.     }
  101.     return YES;
  102. }
  104. - (BOOL)linkProgram:(GLuint)prog
  105. {
  106.     GLint status;
  107.     glLinkProgram(prog);
  109. #if defined(DEBUG)
  110.     GLint logLength;
  111.     glGetProgramiv(prog, GL_INFO_LOG_LENGTH, &logLength);
  112.     if (logLength > 0) {
  113.         GLchar *log = (GLchar *)malloc(logLength);
  114.         glGetProgramInfoLog(prog, logLength, &logLength, log);
  115.         NSLog(@"Program link log:\n%s", log);
  116.         free(log);
  117.     }
  118. #endif
  120.     glGetProgramiv(prog, GL_LINK_STATUS, &status);
  121.     if (status == 0) {
  122.         return NO;
  123.     }
  125.     return YES;
  126. }

渲染处理

  1. - (void)renderPixelBuffer:(CVPixelBufferRef)destinationPixelBuffer
  2.     foregroundPixelBuffer:(CVPixelBufferRef)foregroundPixelBuffer
  3.     backgroundPixelBuffer:(CVPixelBufferRef)backgroundPixelBuffer
  4.               tweenFactor:(float)tween {
  5.     if (!foregroundPixelBuffer || !backgroundPixelBuffer) {
  6.         return;
  7.     }
  8.     [EAGLContext setCurrentContext:self.currentContext];
  10.     CVOpenGLESTextureRef foregroundTexture = [self textureForPixelBuffer:foregroundPixelBuffer];
  11.     CVOpenGLESTextureRef backgroundTexture = [self textureForPixelBuffer:backgroundPixelBuffer];
  12.     CVOpenGLESTextureRef destTexture = [self textureForPixelBuffer:destinationPixelBuffer];
  14.     glUseProgram(self.program);
  16.     glBindFramebuffer(GL_FRAMEBUFFER, self.offscreenBufferHandle);
  17.     glViewport(0, 0, (int)CVPixelBufferGetWidth(destinationPixelBuffer), (int)CVPixelBufferGetHeight(destinationPixelBuffer));
  19.     // 第一个纹理
  20.     glActiveTexture(GL_TEXTURE0);
  21.     glBindTexture(CVOpenGLESTextureGetTarget(foregroundTexture), CVOpenGLESTextureGetName(foregroundTexture));
  22.     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
  23.     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
  24.     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
  25.     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
  27.     // 第二个纹理
  28.     glActiveTexture(GL_TEXTURE1);
  29.     glBindTexture(CVOpenGLESTextureGetTarget(backgroundTexture), CVOpenGLESTextureGetName(backgroundTexture));
  30.     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
  31.     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
  32.     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
  33.     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
  35.     // Attach the destination texture as a color attachment to the off screen frame buffer
  36.     glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, CVOpenGLESTextureGetTarget(destTexture), CVOpenGLESTextureGetName(destTexture), 0);
  37.     if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE) {
  38.         NSLog(@"Failed to make complete framebuffer object %x", glCheckFramebufferStatus(GL_FRAMEBUFFER));
  39.         return;
  40.     }
  42.     // clear
  43.     glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
  44.     glClear(GL_COLOR_BUFFER_BIT);
  46.     // 顶点
  47.     GLfloat quadVertexData [] = {
  48.         -1.0, 1.0,
  49.         1.0, 1.0,
  50.         -1.0, -1.0,
  51.         1.0, -1.0,
  52.     };
  54.     // texture data varies from 0 -> 1, whereas vertex data varies from -1 -> 1
  55.     GLfloat quadTextureData [] = {
  56.         0.5 + quadVertexData[0]/2, 0.5 + quadVertexData[1]/2,
  57.         0.5 + quadVertexData[2]/2, 0.5 + quadVertexData[3]/2,
  58.         0.5 + quadVertexData[4]/2, 0.5 + quadVertexData[5]/2,
  59.         0.5 + quadVertexData[6]/2, 0.5 + quadVertexData[7]/2,
  60.     };
  62.     // 应用第一个纹理
  63.     glUniform1i(uniforms[UNIFORM], 0);
  65.     // 设置顶点数据
  66.     glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, quadVertexData);
  67.     glEnableVertexAttribArray(ATTRIB_VERTEX);
  69.     glVertexAttribPointer(ATTRIB_TEXCOORD, 2, GL_FLOAT, 0, 0, quadTextureData);
  70.     glEnableVertexAttribArray(ATTRIB_TEXCOORD);
  72.     // 启用混合模式
  73.     glEnable(GL_BLEND);
  75.     // 混合模式为，全源
  76.     glBlendFunc(GL_ONE, GL_ZERO);
  78.     // 绘制前景
  79.     glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
  81.     // 应用第二个纹理
  82.     glUniform1i(uniforms[UNIFORM], 1);
  84.     // 设置顶点数据
  85.     glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, quadVertexData);
  86.     glEnableVertexAttribArray(ATTRIB_VERTEX);
  88.     glVertexAttribPointer(ATTRIB_TEXCOORD, 2, GL_FLOAT, 0, 0, quadTextureData);
  89.     glEnableVertexAttribArray(ATTRIB_TEXCOORD);
  91.     // 混合模式绘制背景
  92.     // GL_CONSTANT_ALPHA 采用 glBlendColor 中的 alpha 值
  93.     glBlendColor(0, 0, 0, tween);
  94.     glBlendFunc(GL_CONSTANT_ALPHA, GL_ONE_MINUS_CONSTANT_ALPHA);
  96.     // 绘制背景
  97.     glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
  98.     glFlush();
  100.     // Periodic texture cache flush every frame
  101.     CVOpenGLESTextureCacheFlush(self.videoTextureCache, 0);
  103.     [EAGLContext setCurrentContext:nil];
  104. }
  106. - (CVOpenGLESTextureRef)textureForPixelBuffer:(CVPixelBufferRef)pixelBuffer {
  107.     CVOpenGLESTextureRef texture = NULL;
  108.     CVReturn err;
  110.     if (!_videoTextureCache) {
  111.         NSLog(@"No video texture cache");
  112.         return texture;
  113.     }
  115.     // Periodic texture cache flush every frame
  116.     CVOpenGLESTextureCacheFlush(_videoTextureCache, 0);
  118.     // CVOpenGLTextureCacheCreateTextureFromImage will create GL texture optimally from CVPixelBufferRef.
  119.     err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
  120.                                                        _videoTextureCache,
  121.                                                        pixelBuffer,
  122.                                                        NULL,
  123.                                                        GL_TEXTURE_2D,
  124.                                                        GL_RGBA,
  125.                                                        (int)CVPixelBufferGetWidth(pixelBuffer),
  126.                                                        (int)CVPixelBufferGetHeight(pixelBuffer),
  127.                                                        GL_RGBA,
  128.                                                        GL_UNSIGNED_BYTE,
  129.                                                        0,
  130.                                                        &texture);
  132.     if (!texture || err) {
  133.         NSLog(@"Error at creating luma texture using CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
  134.     }
  135.     return texture;
  136. }

Metal 转场

shader

  1. #include <metal_stdlib>
  2. #import "ShaderTypes.h"
  4. using namespace metal;
  6. typedef struct
  7. {
  8.     float4 clipSpacePosition [[ position ]]; // position 修饰符表示这个是顶点
  9.     float2 textureCoordinate;
  10.     float  factor;
  11. } RasterizerData;
  13. vertex RasterizerData vertexShader(uint vertexID [[ vertex_id ]],
  14.                                    constant Vertex *vertexArray [[ buffer(VertexInputIndexVertices) ]]) {
  15.     RasterizerData out;
  16.     out.clipSpacePosition = float4(vertexArray[vertexID].position, 0.0, 1.0);
  17.     out.textureCoordinate = vertexArray[vertexID].textureCoordinate;
  18.     return out;
  19. }
  21. fragment float4 samplingShader(RasterizerData input [[stage_in]],
  22.                                texture2d<float> foregroundTexture [[ texture(FragmentTextureIndexForeground) ]],
  23.                                texture2d<float> backgroundTexture [[ texture(FragmentTextureIndexbakcground) ]],
  24.                                constant float &factor [[ buffer(FragmentInputIndexFactor) ]]) {
  25.     constexpr sampler textureSampler (mag_filter::linear,
  26.                                       min_filter::linear);
  27.     float3 forgroundColor = foregroundTexture.sample(textureSampler, input.textureCoordinate).rgb;
  28.     float3 backgroundColor = backgroundTexture.sample(textureSampler, input.textureCoordinate).rgb;
  30.     float3 color = forgroundColor * (1 - factor) + backgroundColor * factor;
  31.     return float4(color, 1.0);
  32. }

设置 shader 着色器

  1. - (void)setupPipeline {
  2.     id<MTLLibrary> defaultLibrary = [self.device newDefaultLibrary];
  3.     id<MTLFunction> vertexFunction = [defaultLibrary newFunctionWithName:@"vertexShader"];
  4.     id<MTLFunction> fragmentFunction = [defaultLibrary newFunctionWithName:@"samplingShader"];
  6.     MTLRenderPipelineDescriptor *pipelineDescriptor = [[MTLRenderPipelineDescriptor alloc] init];
  7.     pipelineDescriptor.vertexFunction = vertexFunction;
  8.     pipelineDescriptor.fragmentFunction = fragmentFunction;
  9.     pipelineDescriptor.colorAttachments[0].pixelFormat = MTLPixelFormatBGRA8Unorm;
  11.     self.pipelineState = [self.device newRenderPipelineStateWithDescriptor:pipelineDescriptor error:NULL];
  12.     self.commandQueue = [self.device newCommandQueue];
  13. }
  15. - (void)setupVertex {
  16.     Vertex quardVertices[] =
  17.     {   // 顶点坐标，分别是x、y    纹理坐标，x、y；
  18.         { {  1.0, -1.0 },  { 1.f, 1.f } },
  19.         { { -1.0, -1.0 },  { 0.f, 1.f } },
  20.         { { -1.0,  1.0 },  { 0.f, 0.f } },
  22.         { {  1.0, -1.0 },  { 1.f, 1.f } },
  23.         { { -1.0,  1.0 },  { 0.f, 0.f } },
  24.         { {  1.0,  1.0 },  { 1.f, 0.f } },
  25.     };
  26.     self.vertices = [self.device newBufferWithBytes:quardVertices
  27.                                              length:sizeof(quardVertices)
  28.                                             options:MTLResourceStorageModeShared];
  29.     self.numVertices = sizeof(quardVertices) / sizeof(Vertex);
  30. }

渲染处理

  1. - (void)renderPixelBuffer:(CVPixelBufferRef)destinationPixelBuffer
  2.     foregroundPixelBuffer:(CVPixelBufferRef)foregroundPixelBuffer
  3.     backgroundPixelBuffer:(CVPixelBufferRef)backgroundPixelBuffer
  4.               tweenFactor:(float)tween {
  5.     id<MTLTexture> destinationTexture = [self textureWithCVPixelBuffer:destinationPixelBuffer];
  6.     id<MTLTexture> foregroundTexture = [self textureWithCVPixelBuffer:foregroundPixelBuffer];
  7.     id<MTLTexture> backgroundTexture = [self textureWithCVPixelBuffer:backgroundPixelBuffer];
  9.     id<MTLCommandBuffer> commandBuffer = [self.commandQueue commandBuffer];
  10.     MTLRenderPassDescriptor *renderDescriptor = [MTLRenderPassDescriptor renderPassDescriptor];
  11.     renderDescriptor.colorAttachments[0].clearColor = MTLClearColorMake(0, 0, 0, 1);
  12.     renderDescriptor.colorAttachments[0].storeAction = MTLStoreActionStore;
  13.     renderDescriptor.colorAttachments[0].loadAction = MTLLoadActionClear;
  14.     renderDescriptor.colorAttachments[0].texture = destinationTexture;
  16.     id<MTLRenderCommandEncoder> renderEncoder = [commandBuffer renderCommandEncoderWithDescriptor:renderDescriptor];
  17.     [renderEncoder setRenderPipelineState:self.pipelineState];
  18.     [renderEncoder setVertexBuffer:self.vertices offset:0 atIndex:VertexInputIndexVertices];
  19.     [renderEncoder setFragmentTexture:foregroundTexture atIndex:FragmentTextureIndexForeground];
  20.     [renderEncoder setFragmentTexture:backgroundTexture atIndex:FragmentTextureIndexbakcground];
  21.     [renderEncoder setFragmentBytes:&tween length:sizeof(tween) atIndex:FragmentInputIndexFactor];
  22.     [renderEncoder drawPrimitives:MTLPrimitiveTypeTriangle
  23.                       vertexStart:0
  24.                       vertexCount:self.numVertices]; // 绘制
  26.     [renderEncoder endEncoding]; // 结束
  27.     [commandBuffer commit];
  28. }
  30. - (id<MTLTexture>)textureWithCVPixelBuffer:(CVPixelBufferRef)pixelBuffer {
  31.     if (pixelBuffer == NULL) {
  32.         return nil;
  33.     }
  34.     id<MTLTexture> texture = nil;
  35.     CVMetalTextureRef metalTextureRef = NULL;
  36.     size_t width = CVPixelBufferGetWidth(pixelBuffer);
  37.     size_t height = CVPixelBufferGetHeight(pixelBuffer);
  38.     MTLPixelFormat pixelFormat = MTLPixelFormatBGRA8Unorm;
  39.     CVReturn status = CVMetalTextureCacheCreateTextureFromImage(NULL,
  40.                                                                 _textureCache,
  41.                                                                 pixelBuffer,
  42.                                                                 NULL,
  43.                                                                 pixelFormat,
  44.                                                                 width,
  45.                                                                 height,
  46.                                                                 0,
  47.                                                                 &metalTextureRef);
  48.     if (status == kCVReturnSuccess) {
  49.         texture = CVMetalTextureGetTexture(metalTextureRef);
  50.         CFRelease(metalTextureRef);
  51.     }
  52.     return texture;
  53. }

如果你正在跳槽或者正准备跳槽不妨动动小手，添加一下咱们的交流群1012951431来获取一份详细的大厂面试资料为你的跳槽多添一份保障。

更多精彩分享